Sep 17 17:33:55.487263 ip-10-0-133-194 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:33:58.975383 ip-10-0-133-194 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:58.975383 ip-10-0-133-194 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:33:58.975383 ip-10-0-133-194 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:58.975383 ip-10-0-133-194 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:33:58.975383 ip-10-0-133-194 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:58.976229 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.975450    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:33:58.978567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978550    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:58.978567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978565    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978570    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978573    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978576    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978582    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978586    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978590    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978594    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978597    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978600    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978604    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978607    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978610    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978612    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978615    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978618    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978620    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978623    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978625    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978628    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:58.978629 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978631    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978634    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978637    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978639    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978642    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978645    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978647    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978650    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978653    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978655    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978658    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978660    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978663    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978667    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978670    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978673    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978676    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978679    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978682    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:58.979150 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978684    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978686    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978689    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978692    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978695    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978697    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978700    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978731    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978735    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978738    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978741    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978744    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978747    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978749    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978752    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978756    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978759    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978762    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978765    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:58.979610 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978768    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978770    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978773    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978776    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978778    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978781    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978784    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978786    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978789    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978791    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978794    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978797    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978800    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978802    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978805    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978807    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978810    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978813    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978815    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978818    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:58.980092 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978821    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978823    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978826    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978829    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978831    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978834    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978837    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978840    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978843    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978845    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978848    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978850    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978853    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978855    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978858    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978860    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978863    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978865    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978868    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.978871    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:58.980614 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979529    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979536    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979539    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979542    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979546    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979549    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979552    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979554    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979557    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979559    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979562    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979566    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979568    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979571    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979573    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979576    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979578    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979581    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979583    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979586    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:58.981109 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979588    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979591    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979594    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979596    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979599    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979601    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979604    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979607    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979609    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979612    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979615    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979618    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979621    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979625    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979629    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979632    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979635    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979638    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979641    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979643    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:58.981606 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979646    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979648    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979651    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979653    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979656    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979659    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979661    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979664    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979667    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979669    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979671    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979674    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979677    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979680    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979682    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979685    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979687    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979689    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979692    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979695    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:58.982159 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979699    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979701    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979704    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979706    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979709    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979711    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979714    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979716    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979719    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979722    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979725    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979727    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979729    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979732    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979734    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979737    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979739    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979742    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979745    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:58.982661 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979748    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979751    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979754    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979757    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979760    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979763    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979766    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979768    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979771    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979775    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979778    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979782    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979784    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979788    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979791    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979793    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979796    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979798    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979801    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:58.983147 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.979803    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980164    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980169    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980172    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980175    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980178    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980181    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980183    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980186    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980189    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980191    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980194    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980197    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980199    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980202    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980205    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980208    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980211    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980214    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980216    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:58.983616 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980219    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980222    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980225    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980228    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980231    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980234    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980237    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980240    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980242    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980245    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980248    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980251    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980253    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980256    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980259    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980261    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980264    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980267    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980269    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:58.984102 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980272    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980275    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980277    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980280    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980282    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980285    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980289    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980292    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980295    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980298    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980301    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980304    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980307    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980309    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980312    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980315    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980317    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980320    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980323    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:58.984572 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980326    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980329    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980331    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980334    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980337    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980339    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980341    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980344    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980347    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980349    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980352    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980354    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980357    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980360    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980364    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980367    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980370    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980373    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980375    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980378    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:58.985049 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980381    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980384    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980387    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980390    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980393    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980395    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980398    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980401    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980404    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980407    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980410    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980412    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980417    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980420    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980423    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980425    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980428    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980430    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980433    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980435    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:58.985574 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980438    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.980440    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980507    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980552    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980558    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980564    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980573    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980577    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980581    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980585    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980588    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980591    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980595    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980598    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980601    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980604    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980607    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980610    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980613    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980616    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980621    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980624    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980627    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980631    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:33:58.986083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980635    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980640    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980643    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980646    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980649    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980652    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980655    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980658    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980662    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980665    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980669    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980672    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980675    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980678    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980681    2581 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980685    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980689    2581 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980692    2581 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980695    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980698    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980701    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980704    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980707    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980710    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980726    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:33:58.986661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980731    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980734    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980737    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980740    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980743    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980746    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980749    2581 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980754    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980757    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980760    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980763    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980766    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980769    2581 flags.go:64] FLAG: --help="false"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980772    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-133-194.ec2.internal"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980775    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980778    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980781    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980785    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980789    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980793    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980796    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980799    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980802    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:33:58.987273 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980805    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980809    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980811    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980814    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980817    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980820    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980823    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980826    2581 flags.go:64] FLAG: --lock-file=""
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980829    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980832    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980835    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980840    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980843    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980846    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980849    2581 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980853    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980857    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980860    2581 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980863    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980867    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980870    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980874    2581 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980877    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980880    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980883    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:33:58.987868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980886    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980889    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980892    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980898    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980904    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980907    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980911    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980914    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980917    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980922    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980925    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980928    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980931    2581 flags.go:64] FLAG: --port="10250"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980934    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980938    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08267b58d7ab8aeab"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980941    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980944    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980947    2581 flags.go:64] FLAG: --register-node="true"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980950    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980953    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980956    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980959    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980974    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980979    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980982    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:33:58.988492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980985    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980988    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980991    2581 flags.go:64] FLAG: --runonce="false"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980994    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.980997    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981001    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981003    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981006    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981009    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981013    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981016    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981020    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981023    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981026    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981029    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981032    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981036    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981038    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981044    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981047    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981049    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981054    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981057    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981060    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981063    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:33:58.989427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981066    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981069    2581 flags.go:64] FLAG: --v="2"
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981073    2581 flags.go:64] FLAG: --version="false"
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981076    2581 flags.go:64] FLAG: --vmodule=""
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981080    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.981084    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981198    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981202    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981206    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981209    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981212    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981215    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981217    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981220    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981223    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981225    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981228    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981231    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981235    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981238    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:58.990199 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981241    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981244    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981248    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981252    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981255    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981258    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981261    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981263    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981266    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981269    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981271    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981274    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981276    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981279    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981282    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981284    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981287    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981292    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981294    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:58.990700 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981297    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981300    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981302    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981305    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981307    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981310    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981313    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981315    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981318    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981321    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981323    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981327    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981330    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981332    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981335    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981337    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981340    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981343    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981346    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981349    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:58.991173 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981352    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981354    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981357    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981360    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981362    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981365    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981368    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981370    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981373    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981376    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981380    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981382    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981385    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981388    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981390    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981393    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981396    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981398    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981402    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981406    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:58.991675 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981408    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981412    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981415    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981419    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981422    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981425    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981428    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981431    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981434    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981436    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981439    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981442    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981445    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981447    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981450    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981454    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981457    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981460    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981462    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:58.992181 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981465    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981468    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981470    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981475    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981478    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981480    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.981483    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.982201    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.990751    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.990766    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.990786    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.990792    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990849    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990853    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990858    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:58.992695 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990861    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990863    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990866    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990869    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990872    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990875    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990878    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990881    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990883    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990886    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990889    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990891    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990894    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990896    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990899    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990902    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990904    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990907    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990910    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990912    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:58.993089 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990916    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990918    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990921    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990924    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990926    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990929    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990933    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990935    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990940    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990943    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990946    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990949    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990952    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990954    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990957    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990960    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990974    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990977    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990980    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:58.993567 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990983    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990985    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990989    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990993    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.990996    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991000    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991003    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991006    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991008    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991012    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991014    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991018    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991021    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991024    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991027    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991029    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991032    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991035    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991037    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:58.994053 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991040    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991043    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991046    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991049    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991052    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991054    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991057    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991060    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991062    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991065    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991068    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991070    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991073    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991075    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991078    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991081    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991083    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991086    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991090    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991092    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:58.994555 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991095    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991098    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991100    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991103    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991106    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991108    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991111    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991114    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991116    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991119    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991121    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991124    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991127    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991131    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991135    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991139    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991141    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:58.995101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991144    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.991150    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991260    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991265    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991268    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991271    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991274    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991277    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991280    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991283    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991285    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991288    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991291    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991294    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991297    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:58.995547 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991300    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991303    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991306    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991308    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991311    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991313    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991316    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991319    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991322    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991324    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991327    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991329    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991332    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991335    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991338    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991340    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991343    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991346    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991348    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991351    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:58.995926 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991354    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991356    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991359    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991362    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991365    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991369    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991372    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991374    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991377    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991379    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991382    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991385    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991391    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991394    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991397    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991399    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991402    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991404    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991407    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991410    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:58.996432 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991412    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991415    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991417    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991420    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991423    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991425    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991429    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991431    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991434    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991437    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991440    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991444    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991447    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991450    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991453    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991456    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991458    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991461    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991463    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:58.996912 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991466    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991468    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991471    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991474    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991476    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991479    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991485    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991488    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991491    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991494    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991497    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991500    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991503    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991506    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991509    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991511    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991514    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991516    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991519    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:58.997391 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991522    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991524    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991527    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991530    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991532    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991535    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991537    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:33:58.991540    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.991545    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.991663    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.994366    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.995328    2581 server.go:1019] "Starting client certificate rotation"
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.995423    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:58.997839 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.995458    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:58.998207 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.997940    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:58.999138 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:58.999060    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:59.012645 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.012624    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:33:59.017425 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.017408    2581 log.go:25] "Validated CRI v1 image API"
Sep 17 17:33:59.018234 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.018217    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:33:59.020541 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.020522    2581 fs.go:135] Filesystem UUIDs: map[1afee08b-b74e-4333-af4c-7bce0e829eae:/dev/nvme0n1p3 6680627f-0f9d-475d-af93-cdf14d06caaf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 17:33:59.020601 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.020539    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:33:59.026078 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.026062    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:33:59.028903 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.028319    2581 manager.go:216] Machine: {Timestamp:2026-09-17 17:33:59.02783235 +0000 UTC m=+1.952304300 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089129 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2961dee60b677d968ba587948f84e4 SystemUUID:ec2961de-e60b-677d-968b-a587948f84e4 BootID:1c1c0dc6-b1f7-45fc-ba9a-5c67ccbbf32f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f1:7a:f8:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f1:7a:f8:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:6e:92:c0:55:d3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:33:59.029026 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.028915    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:33:59.029271 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.029238    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:33:59.032163 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.032135    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:33:59.032292 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.032163    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:33:59.033703 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.033692    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:33:59.033738 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.033705    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:33:59.033738 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.033713    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:59.033738 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.033725    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:59.033818 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.033797    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:33:59.036837 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.036825    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:59.036938 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.036929    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:33:59.036989 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.036959    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:33:59.036989 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.036979    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:33:59.037042 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.036996    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:33:59.037042 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.037006    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:33:59.037042 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.037014    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:33:59.037371 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.037360    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:59.037414 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.037378    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:59.040111 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.040095    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:33:59.042563 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.042549    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:33:59.042623 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.042580    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:33:59.047283 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047263    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:33:59.047348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047293    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:33:59.047348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047302    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:33:59.047348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047315    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:33:59.047348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047325    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:33:59.047348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047333    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:33:59.047490 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047353    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:33:59.047490 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047359    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:33:59.047490 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047368    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:33:59.047490 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047373    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:33:59.047490 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047381    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:33:59.047490 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047386    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:33:59.047490 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047407    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:33:59.047490 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.047412    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:33:59.051337 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.051318    2581 server.go:1268] "Started kubelet"
Sep 17 17:33:59.051417 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.051376    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:33:59.051480 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.051443    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:33:59.051523 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.051494    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:33:59.052729 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.052709    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:33:59.052827 ip-10-0-133-194 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:33:59.054159 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.054141    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dx6tz"
Sep 17 17:33:59.054692 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.054679    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:33:59.054948 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.054926    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:33:59.055052 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.054994    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:33:59.055291 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.055263    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:33:59.057951 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.057931    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:33:59.059107 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.059092    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:33:59.059189 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.059146    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:33:59.059317 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.059293    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.060094 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060075    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:33:59.060094 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060092    2581 factory.go:55] Registering systemd factory
Sep 17 17:33:59.060227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060101    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:33:59.060227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060134    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:33:59.060227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060150    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:33:59.060227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060160    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:33:59.060227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060218    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:33:59.060227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060230    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:33:59.060473 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060369    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dx6tz"
Sep 17 17:33:59.060473 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060376    2581 factory.go:153] Registering CRI-O factory
Sep 17 17:33:59.060473 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060391    2581 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:33:59.060473 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060406    2581 factory.go:103] Registering Raw factory
Sep 17 17:33:59.060473 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.060443    2581 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:33:59.061528 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.061514    2581 manager.go:349] Starting recovery of all containers
Sep 17 17:33:59.062064 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.062040    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:33:59.062293 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.062151    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:33:59.092581 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.092565    2581 manager.go:354] Recovery completed
Sep 17 17:33:59.096473 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.096459    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:59.098222 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.098206    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:59.098290 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.098238    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:59.098290 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.098254    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:59.098785 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.098769    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:33:59.098785 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.098781    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:33:59.098907 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.098783    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:59.098907 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.098799    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:59.099504 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.099486    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:59.099573 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.099515    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:59.099573 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.099528    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:59.099843 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.099832    2581 policy_none.go:49] "None policy: Start"
Sep 17 17:33:59.099880 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.099847    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:33:59.099880 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.099857    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:33:59.101056 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.101045    2581 policy_none.go:47] "Start"
Sep 17 17:33:59.133549 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.133535    2581 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:33:59.142915 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.133567    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:33:59.142915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.133577    2581 server.go:87] "Starting device plugin registration server"
Sep 17 17:33:59.142915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.133807    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:33:59.142915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.133937    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:33:59.142915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.134015    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:33:59.142915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.134100    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:33:59.142915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.134107    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:33:59.142915 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.134628    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:33:59.142915 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.134669    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.173021 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.172992    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:33:59.174166 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.174147    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:33:59.174263 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.174187    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:33:59.174263 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.174211    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:33:59.174263 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.174256    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:33:59.176070 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.176055    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:59.235146 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.235076    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:59.235984 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.235953    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:59.236117 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.235996    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:59.236117 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.236011    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:59.236117 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.236041    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.245292 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.245279    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.245334 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.245299    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-194.ec2.internal\": node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.256679 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.256664    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.274434 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.274415    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"]
Sep 17 17:33:59.274481 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.274475    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:59.275874 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.275858    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:59.275942 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.275887    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:59.275942 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.275896    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:59.277128 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.277117    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:59.277306 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.277264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.277306 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.277289    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:59.277782 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.277767    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:59.277857 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.277797    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:59.277857 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.277812    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:59.278192 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.278179    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:59.278247 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.278202    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:59.278247 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.278211    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:59.278902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.278887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.278996 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.278918    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:59.279618 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.279604    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:59.279697 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.279632    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:59.279697 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.279649    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:59.302521 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.302502    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-194.ec2.internal\" not found" node="ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.306638 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.306623    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-194.ec2.internal\" not found" node="ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.356853 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.356837    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.457148 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.457114    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.461406 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.461390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c0c5f3cddfb693025739ba654373f8f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-194.ec2.internal\" (UID: \"2c0c5f3cddfb693025739ba654373f8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.461458 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.461414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1eb0595c7fa3ed468211ddf54375204f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"1eb0595c7fa3ed468211ddf54375204f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.461458 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.461432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eb0595c7fa3ed468211ddf54375204f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"1eb0595c7fa3ed468211ddf54375204f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.557789 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.557725    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.562082 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.562061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1eb0595c7fa3ed468211ddf54375204f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"1eb0595c7fa3ed468211ddf54375204f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.562167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.562097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eb0595c7fa3ed468211ddf54375204f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"1eb0595c7fa3ed468211ddf54375204f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.562167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.562124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c0c5f3cddfb693025739ba654373f8f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-194.ec2.internal\" (UID: \"2c0c5f3cddfb693025739ba654373f8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.562235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.562187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eb0595c7fa3ed468211ddf54375204f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"1eb0595c7fa3ed468211ddf54375204f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.562235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.562223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c0c5f3cddfb693025739ba654373f8f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-194.ec2.internal\" (UID: \"2c0c5f3cddfb693025739ba654373f8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.562298 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.562187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1eb0595c7fa3ed468211ddf54375204f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"1eb0595c7fa3ed468211ddf54375204f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.606164 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.606146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.610517 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.610356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 17 17:33:59.657824 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.657791    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.758386 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.758338    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.858891 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.858825    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.959411 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:33:59.959360    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:33:59.994693 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.994655    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:33:59.995125 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:33:59.994847    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:00.059984 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:00.059941    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:34:00.073239 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.073213    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:00.085765 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.085737    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:28:59 +0000 UTC" deadline="2028-06-03 20:29:58.654468338 +0000 UTC"
Sep 17 17:34:00.085765 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.085765    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15002h55m58.568706269s"
Sep 17 17:34:00.160179 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:00.160147    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:34:00.255479 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.255454    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:00.260621 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:00.260602    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:34:00.268132 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:00.268105    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0c5f3cddfb693025739ba654373f8f.slice/crio-cb3f5486c1c6892bca2e538b7227d07ae0170785d27a6028886bdfbeba2c2f7d WatchSource:0}: Error finding container cb3f5486c1c6892bca2e538b7227d07ae0170785d27a6028886bdfbeba2c2f7d: Status 404 returned error can't find the container with id cb3f5486c1c6892bca2e538b7227d07ae0170785d27a6028886bdfbeba2c2f7d
Sep 17 17:34:00.272778 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.272762    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:34:00.360704 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:00.360677    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:34:00.432544 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:00.432519    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb0595c7fa3ed468211ddf54375204f.slice/crio-d1880a6258ef785b3f8c42e073cba3f9b6659c52860602e729ad03d2ec1bef50 WatchSource:0}: Error finding container d1880a6258ef785b3f8c42e073cba3f9b6659c52860602e729ad03d2ec1bef50: Status 404 returned error can't find the container with id d1880a6258ef785b3f8c42e073cba3f9b6659c52860602e729ad03d2ec1bef50
Sep 17 17:34:00.461108 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:00.461074    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:34:00.561834 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:00.561796    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 17 17:34:00.631251 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.631223    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:00.660270 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.660228    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 17 17:34:00.671444 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.671424    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:00.673408 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.673386    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 17 17:34:00.682427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.682371    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:00.885355 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:00.885326    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:01.038238 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.038206    2581 apiserver.go:52] "Watching apiserver"
Sep 17 17:34:01.045875 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.045846    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:34:01.046277 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.046248    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-lg444","openshift-multus/network-metrics-daemon-tdmfn","openshift-network-diagnostics/network-check-target-z485x","openshift-network-operator/iptables-alerter-6z2ss","openshift-ovn-kubernetes/ovnkube-node-hbjdm","kube-system/global-pull-secret-syncer-n5zs2","kube-system/konnectivity-agent-9zcnt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc","openshift-cluster-node-tuning-operator/tuned-mdvrr","openshift-dns/node-resolver-c44np","openshift-image-registry/node-ca-frlbs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal","openshift-multus/multus-additional-cni-plugins-vtvtb","kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"]
Sep 17 17:34:01.047761 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.047738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:01.048915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.048888    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.050509 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.050364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:34:01.050737 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.050672    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-crl2l\""
Sep 17 17:34:01.050810 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.050769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:01.050868 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.050836    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:01.050868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.050849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:34:01.051579 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.051396    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:34:01.051579 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.051511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:34:01.051720 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.051698    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:34:01.052073 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.051837    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:34:01.052073 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.051864    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:34:01.052073 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.051872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wghfp\""
Sep 17 17:34:01.052732 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.052709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.054091 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.054070    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.055108 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.055076    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:34:01.055382 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.055357    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:01.055570 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.055550    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gjx7d\""
Sep 17 17:34:01.055667 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.055581    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:01.055878 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.055861    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.055948 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.055929    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:01.057551 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.057503    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.058024 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.058006    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:34:01.058117 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.058101    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:34:01.058117 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.058113    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sh2gh\""
Sep 17 17:34:01.058219 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.058159    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:34:01.058272 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.058220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:34:01.058446 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.058425    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:34:01.058544 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.058487    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:34:01.059776 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.059396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.059871 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.059839    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:01.060057 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.060040    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:01.061218 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.060674    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:01.061218 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.060762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5p685\""
Sep 17 17:34:01.061391 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.061372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:01.061453 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.061439    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:01.061692 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.061672    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:34:01.061959 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.061941    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v5x5h\""
Sep 17 17:34:01.062444 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.062403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.066045 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.065213    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:34:01.066045 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.065278    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:34:01.066045 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.065996    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-47bfd\""
Sep 17 17:34:01.068300 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.067874    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.069300 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.069281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.070047 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-run\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.070150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-run-multus-certs\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.070150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26e0af48-7c11-4be1-b8fb-19d7ee384c6e-iptables-alerter-script\") pod \"iptables-alerter-6z2ss\" (UID: \"26e0af48-7c11-4be1-b8fb-19d7ee384c6e\") " pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.070150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-cni-bin\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.070297 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-ovnkube-script-lib\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.070297 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022180cf-1016-4aa8-ad6a-9b6d20adfeba-tmp-dir\") pod \"node-resolver-c44np\" (UID: \"022180cf-1016-4aa8-ad6a-9b6d20adfeba\") " pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.070297 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxgt\" (UniqueName: \"kubernetes.io/projected/022180cf-1016-4aa8-ad6a-9b6d20adfeba-kube-api-access-rdxgt\") pod \"node-resolver-c44np\" (UID: \"022180cf-1016-4aa8-ad6a-9b6d20adfeba\") " pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.070297 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-system-cni-dir\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.070499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-log-socket\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.070499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.070499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070346    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jzbss\""
Sep 17 17:34:01.070499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070354    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:34:01.070499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-conf-dir\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.070499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4g9\" (UniqueName: \"kubernetes.io/projected/26e0af48-7c11-4be1-b8fb-19d7ee384c6e-kube-api-access-gb4g9\") pod \"iptables-alerter-6z2ss\" (UID: \"26e0af48-7c11-4be1-b8fb-19d7ee384c6e\") " pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.070499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:01.070499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-systemd\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.070868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-var-lib-kubelet\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.070868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-var-lib-cni-bin\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.070868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-run-openvswitch\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.070868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c87361b4-40c1-45cd-a813-271dfe0802d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.070868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mx9\" (UniqueName: \"kubernetes.io/projected/c87361b4-40c1-45cd-a813-271dfe0802d4-kube-api-access-48mx9\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.070868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-sysctl-d\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.070868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070687    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:34:01.070868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:34:01.070868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-tuned\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-etc-kubernetes\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/022180cf-1016-4aa8-ad6a-9b6d20adfeba-hosts-file\") pod \"node-resolver-c44np\" (UID: \"022180cf-1016-4aa8-ad6a-9b6d20adfeba\") " pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c87361b4-40c1-45cd-a813-271dfe0802d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.070992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/243ab975-ee79-47da-9259-405c837db145-dbus\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-lib-modules\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-cni-dir\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2lzw\" (UniqueName: \"kubernetes.io/projected/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-kube-api-access-j2lzw\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-kubelet\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-cni-netd\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-modprobe-d\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-host\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-slash\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-etc-openvswitch\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99jj\" (UniqueName: \"kubernetes.io/projected/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-kube-api-access-k99jj\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.071281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e-konnectivity-ca\") pod \"konnectivity-agent-9zcnt\" (UID: \"d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e\") " pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-run-ovn\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-node-log\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4613160e-a616-4367-bf76-88d4ef9ae7e0-tmp\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-var-lib-kubelet\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-run-systemd\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-kubernetes\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zch6g\" (UniqueName: \"kubernetes.io/projected/4613160e-a616-4367-bf76-88d4ef9ae7e0-kube-api-access-zch6g\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26e0af48-7c11-4be1-b8fb-19d7ee384c6e-host-slash\") pod \"iptables-alerter-6z2ss\" (UID: \"26e0af48-7c11-4be1-b8fb-19d7ee384c6e\") " pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-env-overrides\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-ovn-node-metrics-cert\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c87361b4-40c1-45cd-a813-271dfe0802d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-daemon-config\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-ovnkube-config\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-systemd-units\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.072023 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-run-netns\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-sysctl-conf\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-cnibin\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-socket-dir-parent\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-run-netns\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-var-lib-cni-multus\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rk5b\" (UniqueName: \"kubernetes.io/projected/3b795638-8c73-4961-9d78-d850444de74c-kube-api-access-9rk5b\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071867    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071880    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c87361b4-40c1-45cd-a813-271dfe0802d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-sysconfig\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-run-k8s-cni-cncf-io\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.071992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e-agent-certs\") pod \"konnectivity-agent-9zcnt\" (UID: \"d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e\") " pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87361b4-40c1-45cd-a813-271dfe0802d4-system-cni-dir\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c87361b4-40c1-45cd-a813-271dfe0802d4-cnibin\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-hostroot\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/243ab975-ee79-47da-9259-405c837db145-kubelet-config\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.072749 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-psdff\""
Sep 17 17:34:01.073716 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-sys\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.073716 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072246    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:34:01.073716 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-cni-binary-copy\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.073716 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:01.073716 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.072308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-var-lib-openvswitch\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.078718 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.078695    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:01.097865 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.097833    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9256n"
Sep 17 17:34:01.105386 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.105360    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9256n"
Sep 17 17:34:01.161323 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.161280    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-var-lib-cni-multus\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rk5b\" (UniqueName: \"kubernetes.io/projected/3b795638-8c73-4961-9d78-d850444de74c-kube-api-access-9rk5b\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c87361b4-40c1-45cd-a813-271dfe0802d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-sysconfig\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-run-k8s-cni-cncf-io\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e-agent-certs\") pod \"konnectivity-agent-9zcnt\" (UID: \"d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e\") " pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/388ecc81-7714-4160-adbb-e4e68c6f66ef-host\") pod \"node-ca-frlbs\" (UID: \"388ecc81-7714-4160-adbb-e4e68c6f66ef\") " pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/388ecc81-7714-4160-adbb-e4e68c6f66ef-serviceca\") pod \"node-ca-frlbs\" (UID: \"388ecc81-7714-4160-adbb-e4e68c6f66ef\") " pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-sysconfig\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87361b4-40c1-45cd-a813-271dfe0802d4-system-cni-dir\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c87361b4-40c1-45cd-a813-271dfe0802d4-cnibin\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87361b4-40c1-45cd-a813-271dfe0802d4-system-cni-dir\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-hostroot\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-run-k8s-cni-cncf-io\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-hostroot\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.172906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-registration-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-device-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c87361b4-40c1-45cd-a813-271dfe0802d4-cnibin\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-var-lib-cni-multus\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.172952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/243ab975-ee79-47da-9259-405c837db145-kubelet-config\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-sys\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-cni-binary-copy\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/243ab975-ee79-47da-9259-405c837db145-kubelet-config\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-sys\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-var-lib-openvswitch\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-run\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173258    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.173272    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-run-multus-certs\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26e0af48-7c11-4be1-b8fb-19d7ee384c6e-iptables-alerter-script\") pod \"iptables-alerter-6z2ss\" (UID: \"26e0af48-7c11-4be1-b8fb-19d7ee384c6e\") " pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.173831 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.173334    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs podName:3b795638-8c73-4961-9d78-d850444de74c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:01.673314446 +0000 UTC m=+4.597786385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs") pod "network-metrics-daemon-tdmfn" (UID: "3b795638-8c73-4961-9d78-d850444de74c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-cni-bin\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-ovnkube-script-lib\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c87361b4-40c1-45cd-a813-271dfe0802d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022180cf-1016-4aa8-ad6a-9b6d20adfeba-tmp-dir\") pod \"node-resolver-c44np\" (UID: \"022180cf-1016-4aa8-ad6a-9b6d20adfeba\") " pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxgt\" (UniqueName: \"kubernetes.io/projected/022180cf-1016-4aa8-ad6a-9b6d20adfeba-kube-api-access-rdxgt\") pod \"node-resolver-c44np\" (UID: \"022180cf-1016-4aa8-ad6a-9b6d20adfeba\") " pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-system-cni-dir\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-log-socket\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-run\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4bq6\" (UniqueName: \"kubernetes.io/projected/e3d4f629-4cd1-4cc4-b240-1ad49226294d-kube-api-access-d4bq6\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-conf-dir\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-cni-bin\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4g9\" (UniqueName: \"kubernetes.io/projected/26e0af48-7c11-4be1-b8fb-19d7ee384c6e-kube-api-access-gb4g9\") pod \"iptables-alerter-6z2ss\" (UID: \"26e0af48-7c11-4be1-b8fb-19d7ee384c6e\") " pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-socket-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-run-multus-certs\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.174569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-cni-binary-copy\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-conf-dir\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-system-cni-dir\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-var-lib-openvswitch\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173778    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-log-socket\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-systemd\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022180cf-1016-4aa8-ad6a-9b6d20adfeba-tmp-dir\") pod \"node-resolver-c44np\" (UID: \"022180cf-1016-4aa8-ad6a-9b6d20adfeba\") " pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-var-lib-kubelet\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-var-lib-cni-bin\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-systemd\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26e0af48-7c11-4be1-b8fb-19d7ee384c6e-iptables-alerter-script\") pod \"iptables-alerter-6z2ss\" (UID: \"26e0af48-7c11-4be1-b8fb-19d7ee384c6e\") " pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-run-openvswitch\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-var-lib-cni-bin\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-var-lib-kubelet\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.173997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-run-openvswitch\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.175374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcflx\" (UniqueName: \"kubernetes.io/projected/388ecc81-7714-4160-adbb-e4e68c6f66ef-kube-api-access-jcflx\") pod \"node-ca-frlbs\" (UID: \"388ecc81-7714-4160-adbb-e4e68c6f66ef\") " pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c87361b4-40c1-45cd-a813-271dfe0802d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mx9\" (UniqueName: \"kubernetes.io/projected/c87361b4-40c1-45cd-a813-271dfe0802d4-kube-api-access-48mx9\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-sysctl-d\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-tuned\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-etc-kubernetes\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/022180cf-1016-4aa8-ad6a-9b6d20adfeba-hosts-file\") pod \"node-resolver-c44np\" (UID: \"022180cf-1016-4aa8-ad6a-9b6d20adfeba\") " pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c87361b4-40c1-45cd-a813-271dfe0802d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-sysctl-d\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/243ab975-ee79-47da-9259-405c837db145-dbus\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-lib-modules\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-cni-dir\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2lzw\" (UniqueName: \"kubernetes.io/projected/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-kube-api-access-j2lzw\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-kubelet\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.174946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-cni-netd\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-modprobe-d\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-host\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.176186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-slash\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-etc-openvswitch\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99jj\" (UniqueName: \"kubernetes.io/projected/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-kube-api-access-k99jj\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-ovnkube-script-lib\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e-konnectivity-ca\") pod \"konnectivity-agent-9zcnt\" (UID: \"d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e\") " pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/243ab975-ee79-47da-9259-405c837db145-dbus\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-etc-kubernetes\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c87361b4-40c1-45cd-a813-271dfe0802d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/022180cf-1016-4aa8-ad6a-9b6d20adfeba-hosts-file\") pod \"node-resolver-c44np\" (UID: \"022180cf-1016-4aa8-ad6a-9b6d20adfeba\") " pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c87361b4-40c1-45cd-a813-271dfe0802d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-lib-modules\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-cni-netd\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e-konnectivity-ca\") pod \"konnectivity-agent-9zcnt\" (UID: \"d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e\") " pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-etc-openvswitch\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-host\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.176931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-cni-dir\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-kubelet\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.175953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-slash\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.176037    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.176101    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret podName:243ab975-ee79-47da-9259-405c837db145 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:01.676073125 +0000 UTC m=+4.600545076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret") pod "global-pull-secret-syncer-n5zs2" (UID: "243ab975-ee79-47da-9259-405c837db145") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-run-ovn\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-run-ovn\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-modprobe-d\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-node-log\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4613160e-a616-4367-bf76-88d4ef9ae7e0-tmp\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-var-lib-kubelet\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-node-log\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-run-systemd\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-kubernetes\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zch6g\" (UniqueName: \"kubernetes.io/projected/4613160e-a616-4367-bf76-88d4ef9ae7e0-kube-api-access-zch6g\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26e0af48-7c11-4be1-b8fb-19d7ee384c6e-host-slash\") pod \"iptables-alerter-6z2ss\" (UID: \"26e0af48-7c11-4be1-b8fb-19d7ee384c6e\") " pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-env-overrides\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-ovn-node-metrics-cert\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c87361b4-40c1-45cd-a813-271dfe0802d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-daemon-config\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-ovnkube-config\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-sys-fs\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-systemd-units\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-run-netns\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-sysctl-conf\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-cnibin\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-socket-dir-parent\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.176881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-run-netns\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.177029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-run-netns\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.177094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-host-var-lib-kubelet\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.177183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-run-systemd\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.177249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-kubernetes\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.177551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26e0af48-7c11-4be1-b8fb-19d7ee384c6e-host-slash\") pod \"iptables-alerter-6z2ss\" (UID: \"26e0af48-7c11-4be1-b8fb-19d7ee384c6e\") " pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.177879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.177689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-tuned\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.178597 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.177999    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-env-overrides\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.178951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.178929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-systemd-units\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.179072 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.179048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4613160e-a616-4367-bf76-88d4ef9ae7e0-etc-sysctl-conf\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.179151 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.179135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-host-run-netns\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.179574 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.179550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-daemon-config\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.179659 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.179576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-ovnkube-config\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.179659 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.179625    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c87361b4-40c1-45cd-a813-271dfe0802d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.179768 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.179672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-cnibin\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.179768 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.179674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-multus-socket-dir-parent\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.181398 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.179892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e-agent-certs\") pod \"konnectivity-agent-9zcnt\" (UID: \"d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e\") " pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:01.182347 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.182328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4613160e-a616-4367-bf76-88d4ef9ae7e0-tmp\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.183077 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.182925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-ovn-node-metrics-cert\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.183077 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.183026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rk5b\" (UniqueName: \"kubernetes.io/projected/3b795638-8c73-4961-9d78-d850444de74c-kube-api-access-9rk5b\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:01.184787 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.184692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" event={"ID":"1eb0595c7fa3ed468211ddf54375204f","Type":"ContainerStarted","Data":"d1880a6258ef785b3f8c42e073cba3f9b6659c52860602e729ad03d2ec1bef50"}
Sep 17 17:34:01.186260 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.186234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" event={"ID":"2c0c5f3cddfb693025739ba654373f8f","Type":"ContainerStarted","Data":"cb3f5486c1c6892bca2e538b7227d07ae0170785d27a6028886bdfbeba2c2f7d"}
Sep 17 17:34:01.191079 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.191060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4g9\" (UniqueName: \"kubernetes.io/projected/26e0af48-7c11-4be1-b8fb-19d7ee384c6e-kube-api-access-gb4g9\") pod \"iptables-alerter-6z2ss\" (UID: \"26e0af48-7c11-4be1-b8fb-19d7ee384c6e\") " pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.194030 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.194009    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:01.194030 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.194029    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:01.194172 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.194041    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l4lm4 for pod openshift-network-diagnostics/network-check-target-z485x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:01.194172 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.194087    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4 podName:f67faa62-0294-4089-96d5-14ecee371b01 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:01.694075213 +0000 UTC m=+4.618547156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l4lm4" (UniqueName: "kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4") pod "network-check-target-z485x" (UID: "f67faa62-0294-4089-96d5-14ecee371b01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:01.196223 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.196200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99jj\" (UniqueName: \"kubernetes.io/projected/89ba270d-bdb8-4eb6-ae19-a569f740c1e8-kube-api-access-k99jj\") pod \"ovnkube-node-hbjdm\" (UID: \"89ba270d-bdb8-4eb6-ae19-a569f740c1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.196662 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.196643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2lzw\" (UniqueName: \"kubernetes.io/projected/f8b7d5a8-97c2-4370-9cba-3a0ae2133bff-kube-api-access-j2lzw\") pod \"multus-lg444\" (UID: \"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff\") " pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.196760 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.196675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mx9\" (UniqueName: \"kubernetes.io/projected/c87361b4-40c1-45cd-a813-271dfe0802d4-kube-api-access-48mx9\") pod \"multus-additional-cni-plugins-vtvtb\" (UID: \"c87361b4-40c1-45cd-a813-271dfe0802d4\") " pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.198161 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.198128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxgt\" (UniqueName: \"kubernetes.io/projected/022180cf-1016-4aa8-ad6a-9b6d20adfeba-kube-api-access-rdxgt\") pod \"node-resolver-c44np\" (UID: \"022180cf-1016-4aa8-ad6a-9b6d20adfeba\") " pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.198264 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.198249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zch6g\" (UniqueName: \"kubernetes.io/projected/4613160e-a616-4367-bf76-88d4ef9ae7e0-kube-api-access-zch6g\") pod \"tuned-mdvrr\" (UID: \"4613160e-a616-4367-bf76-88d4ef9ae7e0\") " pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.278102 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-sys-fs\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278102 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/388ecc81-7714-4160-adbb-e4e68c6f66ef-host\") pod \"node-ca-frlbs\" (UID: \"388ecc81-7714-4160-adbb-e4e68c6f66ef\") " pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.278341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/388ecc81-7714-4160-adbb-e4e68c6f66ef-serviceca\") pod \"node-ca-frlbs\" (UID: \"388ecc81-7714-4160-adbb-e4e68c6f66ef\") " pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.278341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-registration-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-sys-fs\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-device-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-device-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278683 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4bq6\" (UniqueName: \"kubernetes.io/projected/e3d4f629-4cd1-4cc4-b240-1ad49226294d-kube-api-access-d4bq6\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278683 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-socket-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278683 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcflx\" (UniqueName: \"kubernetes.io/projected/388ecc81-7714-4160-adbb-e4e68c6f66ef-kube-api-access-jcflx\") pod \"node-ca-frlbs\" (UID: \"388ecc81-7714-4160-adbb-e4e68c6f66ef\") " pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.278683 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278683 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278625    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.278683 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/388ecc81-7714-4160-adbb-e4e68c6f66ef-host\") pod \"node-ca-frlbs\" (UID: \"388ecc81-7714-4160-adbb-e4e68c6f66ef\") " pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.278683 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-registration-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.279065 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/388ecc81-7714-4160-adbb-e4e68c6f66ef-serviceca\") pod \"node-ca-frlbs\" (UID: \"388ecc81-7714-4160-adbb-e4e68c6f66ef\") " pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.279065 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.278753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3d4f629-4cd1-4cc4-b240-1ad49226294d-socket-dir\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.291227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.291152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4bq6\" (UniqueName: \"kubernetes.io/projected/e3d4f629-4cd1-4cc4-b240-1ad49226294d-kube-api-access-d4bq6\") pod \"aws-ebs-csi-driver-node-fz9hc\" (UID: \"e3d4f629-4cd1-4cc4-b240-1ad49226294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.291227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.291152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcflx\" (UniqueName: \"kubernetes.io/projected/388ecc81-7714-4160-adbb-e4e68c6f66ef-kube-api-access-jcflx\") pod \"node-ca-frlbs\" (UID: \"388ecc81-7714-4160-adbb-e4e68c6f66ef\") " pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.362253 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.362212    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:01.372171 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.372145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vtvtb"
Sep 17 17:34:01.383117 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.383091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6z2ss"
Sep 17 17:34:01.388941 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.388919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:01.396780 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.396761    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mdvrr"
Sep 17 17:34:01.404454 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.404435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lg444"
Sep 17 17:34:01.412240 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.412200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c44np"
Sep 17 17:34:01.420132 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.420108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-frlbs"
Sep 17 17:34:01.426299 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.426230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc"
Sep 17 17:34:01.681148 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.681080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:01.681148 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.681136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:01.681325 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.681234    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:01.681325 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.681290    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs podName:3b795638-8c73-4961-9d78-d850444de74c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.681275421 +0000 UTC m=+5.605747363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs") pod "network-metrics-daemon-tdmfn" (UID: "3b795638-8c73-4961-9d78-d850444de74c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:01.681325 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.681234    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:01.681452 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.681356    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret podName:243ab975-ee79-47da-9259-405c837db145 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.681342736 +0000 UTC m=+5.605814688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret") pod "global-pull-secret-syncer-n5zs2" (UID: "243ab975-ee79-47da-9259-405c837db145") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:01.781427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:01.781396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:01.781604 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.781588    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:01.781656 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.781609    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:01.781656 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.781619    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l4lm4 for pod openshift-network-diagnostics/network-check-target-z485x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:01.781721 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:01.781666    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4 podName:f67faa62-0294-4089-96d5-14ecee371b01 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.781653457 +0000 UTC m=+5.706125394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4lm4" (UniqueName: "kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4") pod "network-check-target-z485x" (UID: "f67faa62-0294-4089-96d5-14ecee371b01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:02.108341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.108297    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:01 +0000 UTC" deadline="2028-05-18 02:41:51.568017493 +0000 UTC"
Sep 17 17:34:02.108341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.108336    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14601h7m49.45968531s"
Sep 17 17:34:02.128101 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:02.128075    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022180cf_1016_4aa8_ad6a_9b6d20adfeba.slice/crio-587273d1badd9b2c70d6537372b7bc11823c21a345d2c845b6b0ddd1dc05db01 WatchSource:0}: Error finding container 587273d1badd9b2c70d6537372b7bc11823c21a345d2c845b6b0ddd1dc05db01: Status 404 returned error can't find the container with id 587273d1badd9b2c70d6537372b7bc11823c21a345d2c845b6b0ddd1dc05db01
Sep 17 17:34:02.129156 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:02.129120    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87361b4_40c1_45cd_a813_271dfe0802d4.slice/crio-0d960d7a21c93aed79931cd997c49523174b16478aeba9162c47435812dedc6e WatchSource:0}: Error finding container 0d960d7a21c93aed79931cd997c49523174b16478aeba9162c47435812dedc6e: Status 404 returned error can't find the container with id 0d960d7a21c93aed79931cd997c49523174b16478aeba9162c47435812dedc6e
Sep 17 17:34:02.129903 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:02.129685    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4613160e_a616_4367_bf76_88d4ef9ae7e0.slice/crio-12d598cff950e6fb9644f8b23936c463cf93319294c7d4f30b27f78c6edd0542 WatchSource:0}: Error finding container 12d598cff950e6fb9644f8b23936c463cf93319294c7d4f30b27f78c6edd0542: Status 404 returned error can't find the container with id 12d598cff950e6fb9644f8b23936c463cf93319294c7d4f30b27f78c6edd0542
Sep 17 17:34:02.131378 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:02.131316    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd765f4f6_1b2b_4c0a_b8e9_5228eb337b1e.slice/crio-d4658ea1c3f9d0fd0e43bff72119e8604dceb53feb2421230635db84e226ff80 WatchSource:0}: Error finding container d4658ea1c3f9d0fd0e43bff72119e8604dceb53feb2421230635db84e226ff80: Status 404 returned error can't find the container with id d4658ea1c3f9d0fd0e43bff72119e8604dceb53feb2421230635db84e226ff80
Sep 17 17:34:02.142477 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:02.142450    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d4f629_4cd1_4cc4_b240_1ad49226294d.slice/crio-d2c1bc37daf60d1cba756e53223449e721a2ba3a0628a2a6b8bcb3e63f52b9c4 WatchSource:0}: Error finding container d2c1bc37daf60d1cba756e53223449e721a2ba3a0628a2a6b8bcb3e63f52b9c4: Status 404 returned error can't find the container with id d2c1bc37daf60d1cba756e53223449e721a2ba3a0628a2a6b8bcb3e63f52b9c4
Sep 17 17:34:02.143559 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:02.143534    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e0af48_7c11_4be1_b8fb_19d7ee384c6e.slice/crio-8e5462c3327ebe567757b95c9618587b864c24367e3bb3109183cee45b8e662b WatchSource:0}: Error finding container 8e5462c3327ebe567757b95c9618587b864c24367e3bb3109183cee45b8e662b: Status 404 returned error can't find the container with id 8e5462c3327ebe567757b95c9618587b864c24367e3bb3109183cee45b8e662b
Sep 17 17:34:02.144286 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:02.144195    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ba270d_bdb8_4eb6_ae19_a569f740c1e8.slice/crio-6aea4f8d6b4cad9695015260e7e8f90dbdbbf6eb79e787eca248f1d8e452b6de WatchSource:0}: Error finding container 6aea4f8d6b4cad9695015260e7e8f90dbdbbf6eb79e787eca248f1d8e452b6de: Status 404 returned error can't find the container with id 6aea4f8d6b4cad9695015260e7e8f90dbdbbf6eb79e787eca248f1d8e452b6de
Sep 17 17:34:02.145851 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:02.145624    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388ecc81_7714_4160_adbb_e4e68c6f66ef.slice/crio-913ab50199c828076bb2da393ebb65180271e1ea46164e983260943270bdd362 WatchSource:0}: Error finding container 913ab50199c828076bb2da393ebb65180271e1ea46164e983260943270bdd362: Status 404 returned error can't find the container with id 913ab50199c828076bb2da393ebb65180271e1ea46164e983260943270bdd362
Sep 17 17:34:02.146405 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:02.146295    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b7d5a8_97c2_4370_9cba_3a0ae2133bff.slice/crio-11f502a504216ec5351a6f02d94c530c3e34bd534c5725a7e6e69f105352b387 WatchSource:0}: Error finding container 11f502a504216ec5351a6f02d94c530c3e34bd534c5725a7e6e69f105352b387: Status 404 returned error can't find the container with id 11f502a504216ec5351a6f02d94c530c3e34bd534c5725a7e6e69f105352b387
Sep 17 17:34:02.174427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.174405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:02.174549 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:02.174532    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:02.193613 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.193566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6z2ss" event={"ID":"26e0af48-7c11-4be1-b8fb-19d7ee384c6e","Type":"ContainerStarted","Data":"8e5462c3327ebe567757b95c9618587b864c24367e3bb3109183cee45b8e662b"}
Sep 17 17:34:02.196051 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.196029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mdvrr" event={"ID":"4613160e-a616-4367-bf76-88d4ef9ae7e0","Type":"ContainerStarted","Data":"12d598cff950e6fb9644f8b23936c463cf93319294c7d4f30b27f78c6edd0542"}
Sep 17 17:34:02.197177 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.197157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc" event={"ID":"e3d4f629-4cd1-4cc4-b240-1ad49226294d","Type":"ContainerStarted","Data":"d2c1bc37daf60d1cba756e53223449e721a2ba3a0628a2a6b8bcb3e63f52b9c4"}
Sep 17 17:34:02.198239 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.198217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9zcnt" event={"ID":"d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e","Type":"ContainerStarted","Data":"d4658ea1c3f9d0fd0e43bff72119e8604dceb53feb2421230635db84e226ff80"}
Sep 17 17:34:02.199395 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.199374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c44np" event={"ID":"022180cf-1016-4aa8-ad6a-9b6d20adfeba","Type":"ContainerStarted","Data":"587273d1badd9b2c70d6537372b7bc11823c21a345d2c845b6b0ddd1dc05db01"}
Sep 17 17:34:02.200464 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.200438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-frlbs" event={"ID":"388ecc81-7714-4160-adbb-e4e68c6f66ef","Type":"ContainerStarted","Data":"913ab50199c828076bb2da393ebb65180271e1ea46164e983260943270bdd362"}
Sep 17 17:34:02.201752 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.201731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtvtb" event={"ID":"c87361b4-40c1-45cd-a813-271dfe0802d4","Type":"ContainerStarted","Data":"0d960d7a21c93aed79931cd997c49523174b16478aeba9162c47435812dedc6e"}
Sep 17 17:34:02.203128 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.203100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lg444" event={"ID":"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff","Type":"ContainerStarted","Data":"11f502a504216ec5351a6f02d94c530c3e34bd534c5725a7e6e69f105352b387"}
Sep 17 17:34:02.204376 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.204357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerStarted","Data":"6aea4f8d6b4cad9695015260e7e8f90dbdbbf6eb79e787eca248f1d8e452b6de"}
Sep 17 17:34:02.687863 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.687535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:02.687863 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.687868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:02.688179 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:02.687769    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:02.688179 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:02.688062    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs podName:3b795638-8c73-4961-9d78-d850444de74c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:04.688043989 +0000 UTC m=+7.612515940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs") pod "network-metrics-daemon-tdmfn" (UID: "3b795638-8c73-4961-9d78-d850444de74c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:02.688441 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:02.687998    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:02.688548 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:02.688482    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret podName:243ab975-ee79-47da-9259-405c837db145 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:04.688465609 +0000 UTC m=+7.612937553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret") pod "global-pull-secret-syncer-n5zs2" (UID: "243ab975-ee79-47da-9259-405c837db145") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:02.788313 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:02.788264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:02.788472 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:02.788427    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:02.788472 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:02.788449    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:02.788472 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:02.788461    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l4lm4 for pod openshift-network-diagnostics/network-check-target-z485x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:02.788565 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:02.788521    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4 podName:f67faa62-0294-4089-96d5-14ecee371b01 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:04.788502555 +0000 UTC m=+7.712974492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4lm4" (UniqueName: "kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4") pod "network-check-target-z485x" (UID: "f67faa62-0294-4089-96d5-14ecee371b01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:03.108906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:03.108837    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:01 +0000 UTC" deadline="2028-03-22 21:09:07.527135085 +0000 UTC"
Sep 17 17:34:03.108906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:03.108903    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13251h35m4.418235358s"
Sep 17 17:34:03.183703 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:03.183053    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:03.183703 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:03.183173    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:03.183703 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:03.183576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:03.183703 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:03.183658    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:03.236995 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:03.236087    2581 generic.go:356] "Generic (PLEG): container finished" podID="1eb0595c7fa3ed468211ddf54375204f" containerID="2a7aa2eb373266d6c93baea4f8c51899b7c88f52f1f050cbb82fab7c319a2dc4" exitCode=0
Sep 17 17:34:03.236995 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:03.236805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" event={"ID":"1eb0595c7fa3ed468211ddf54375204f","Type":"ContainerDied","Data":"2a7aa2eb373266d6c93baea4f8c51899b7c88f52f1f050cbb82fab7c319a2dc4"}
Sep 17 17:34:03.249684 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:03.249130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" event={"ID":"2c0c5f3cddfb693025739ba654373f8f","Type":"ContainerStarted","Data":"b33d16bd3402451acf9280c1fd7b79f982ad0bc3d153bd0bb41b731a53bb726e"}
Sep 17 17:34:04.175165 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:04.175131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:04.175628 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:04.175282    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:04.265995 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:04.265857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" event={"ID":"1eb0595c7fa3ed468211ddf54375204f","Type":"ContainerStarted","Data":"a5d5c4cd60fd919add9950f406c9b282b1796555192b4535d71761fe088edf75"}
Sep 17 17:34:04.284660 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:04.284597    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" podStartSLOduration=4.284576228 podStartE2EDuration="4.284576228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:03.274625437 +0000 UTC m=+6.199097397" watchObservedRunningTime="2026-09-17 17:34:04.284576228 +0000 UTC m=+7.209048189"
Sep 17 17:34:04.705354 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:04.705314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:04.705550 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:04.705393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:04.705550 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:04.705534    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:04.705659 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:04.705594    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs podName:3b795638-8c73-4961-9d78-d850444de74c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.705574987 +0000 UTC m=+11.630046929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs") pod "network-metrics-daemon-tdmfn" (UID: "3b795638-8c73-4961-9d78-d850444de74c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:04.706016 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:04.705993    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:04.706103 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:04.706052    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret podName:243ab975-ee79-47da-9259-405c837db145 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.706036069 +0000 UTC m=+11.630508019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret") pod "global-pull-secret-syncer-n5zs2" (UID: "243ab975-ee79-47da-9259-405c837db145") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:04.806196 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:04.806160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:04.806400 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:04.806332    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:04.806400 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:04.806348    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:04.806400 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:04.806359    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l4lm4 for pod openshift-network-diagnostics/network-check-target-z485x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:04.806563 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:04.806426    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4 podName:f67faa62-0294-4089-96d5-14ecee371b01 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.806408433 +0000 UTC m=+11.730880378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4lm4" (UniqueName: "kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4") pod "network-check-target-z485x" (UID: "f67faa62-0294-4089-96d5-14ecee371b01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:05.175357 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:05.174726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:05.175357 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:05.174874    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:05.176130 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:05.175920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:05.176130 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:05.176054    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:06.175243 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:06.175205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:06.175438 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:06.175346    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:07.175204 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:07.175167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:07.175411 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:07.175279    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:07.175852 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:07.175702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:07.175852 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:07.175816    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:08.175439 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:08.175397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:08.175627 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:08.175535    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:08.745442 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:08.744588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:08.745442 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:08.744656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:08.745442 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:08.744801    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:08.745442 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:08.744880    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs podName:3b795638-8c73-4961-9d78-d850444de74c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:16.74485984 +0000 UTC m=+19.669331796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs") pod "network-metrics-daemon-tdmfn" (UID: "3b795638-8c73-4961-9d78-d850444de74c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:08.745442 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:08.745357    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:08.745442 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:08.745407    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret podName:243ab975-ee79-47da-9259-405c837db145 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:16.745392959 +0000 UTC m=+19.669864897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret") pod "global-pull-secret-syncer-n5zs2" (UID: "243ab975-ee79-47da-9259-405c837db145") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:08.845681 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:08.845643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:08.845860 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:08.845834    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:08.845860 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:08.845860    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:08.846016 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:08.845874    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l4lm4 for pod openshift-network-diagnostics/network-check-target-z485x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:08.846016 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:08.845945    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4 podName:f67faa62-0294-4089-96d5-14ecee371b01 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:16.845924902 +0000 UTC m=+19.770396864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4lm4" (UniqueName: "kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4") pod "network-check-target-z485x" (UID: "f67faa62-0294-4089-96d5-14ecee371b01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:09.176504 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:09.176389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:09.176655 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:09.176548    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:09.176655 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:09.176602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:09.176760 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:09.176707    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:10.175427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:10.175234    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:10.175427 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:10.175383    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:11.178310 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:11.178278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:11.178729 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:11.178286    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:11.178729 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:11.178400    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:11.178729 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:11.178494    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:12.174601 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:12.174570    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:12.174712 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:12.174689    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:13.175464 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:13.175435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:13.175887 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:13.175554    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:13.175887 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:13.175637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:13.175887 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:13.175758    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:14.174800 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:14.174769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:14.175001 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:14.174904    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:15.175311 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:15.175278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:15.175721 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:15.175403    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:15.175721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:15.175450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:15.175721 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:15.175538    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:16.174879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:16.174848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:16.175103 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:16.174980    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:16.801596 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:16.801557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:16.802046 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:16.801713    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:16.802046 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:16.801719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:16.802046 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:16.801789    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret podName:243ab975-ee79-47da-9259-405c837db145 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:32.801767142 +0000 UTC m=+35.726239097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret") pod "global-pull-secret-syncer-n5zs2" (UID: "243ab975-ee79-47da-9259-405c837db145") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:16.802046 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:16.801808    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:16.802046 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:16.801861    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs podName:3b795638-8c73-4961-9d78-d850444de74c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:32.801845124 +0000 UTC m=+35.726317076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs") pod "network-metrics-daemon-tdmfn" (UID: "3b795638-8c73-4961-9d78-d850444de74c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:16.902797 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:16.902766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:16.902988 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:16.902886    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:16.902988 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:16.902906    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:16.902988 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:16.902921    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l4lm4 for pod openshift-network-diagnostics/network-check-target-z485x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:16.903134 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:16.903012    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4 podName:f67faa62-0294-4089-96d5-14ecee371b01 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:32.902992875 +0000 UTC m=+35.827464825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4lm4" (UniqueName: "kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4") pod "network-check-target-z485x" (UID: "f67faa62-0294-4089-96d5-14ecee371b01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:17.175281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:17.175202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:17.175442 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:17.175202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:17.175442 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:17.175330    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:17.175442 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:17.175393    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:18.174738 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:18.174707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:18.175178 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:18.174844    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:19.175261 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:19.175230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:19.175670 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:19.175311    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:19.175670 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:19.175422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:19.175670 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:19.175536    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:20.174395 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.174372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:20.174484 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:20.174468    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:20.297195 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.297141    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lg444" event={"ID":"f8b7d5a8-97c2-4370-9cba-3a0ae2133bff","Type":"ContainerStarted","Data":"6008fb714fd7bc04d0c9227d79b12c1a2c91ea717eac0d9a84ae1634fc57f4c1"}
Sep 17 17:34:20.300503 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.300485    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:34:20.300873 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.300853    2581 generic.go:356] "Generic (PLEG): container finished" podID="89ba270d-bdb8-4eb6-ae19-a569f740c1e8" containerID="1094d14007fda67497416b42053ad4f862ca20c0a08a1c84428615528fcff88e" exitCode=1
Sep 17 17:34:20.300948 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.300917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerStarted","Data":"f4935700ac8a6a27b5815e028aea379e1e9210ff34b41ea1fcbb88f1707e44a7"}
Sep 17 17:34:20.301028 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.301012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerStarted","Data":"26badd50b63d0d564320bafc2a2cc721afcda056353d4dae6034633078abd513"}
Sep 17 17:34:20.301085 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.301070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerStarted","Data":"5236426dfd5a0ecd77d3abe72b212fc5b31f1be45dc81d924ce7e749e2752883"}
Sep 17 17:34:20.301135 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.301126    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerStarted","Data":"e0682ab270ee669b58eb9d58eeb541f8a7ccd5d9a5611dc635379d7eeedc0100"}
Sep 17 17:34:20.301177 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.301170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerDied","Data":"1094d14007fda67497416b42053ad4f862ca20c0a08a1c84428615528fcff88e"}
Sep 17 17:34:20.301221 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.301203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerStarted","Data":"439e21705c05a05c80ea01d3c3ecdd4c25ff169472f032c49184d10886960e2f"}
Sep 17 17:34:20.302519 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.302497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mdvrr" event={"ID":"4613160e-a616-4367-bf76-88d4ef9ae7e0","Type":"ContainerStarted","Data":"f2ef8fce291a47e44387fd17b1aba8a8dedc0d1f65771c73a3fe29d72cbebd98"}
Sep 17 17:34:20.304074 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.304051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc" event={"ID":"e3d4f629-4cd1-4cc4-b240-1ad49226294d","Type":"ContainerStarted","Data":"9aaf6fa17be0dabd3631b2f24933866e3be99a65b17a5f303a55445aa65054c0"}
Sep 17 17:34:20.305448 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.305429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9zcnt" event={"ID":"d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e","Type":"ContainerStarted","Data":"996bd11d5ee006b330df8d9e75c429e8ae36aa4e5279d308e95e6daf2806586d"}
Sep 17 17:34:20.306822 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.306800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c44np" event={"ID":"022180cf-1016-4aa8-ad6a-9b6d20adfeba","Type":"ContainerStarted","Data":"a704617b81280c9d783dd630a856ca83014539c0b85233e9e1718e33d4c70013"}
Sep 17 17:34:20.308260 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.308241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-frlbs" event={"ID":"388ecc81-7714-4160-adbb-e4e68c6f66ef","Type":"ContainerStarted","Data":"0608bc08a1335b4a3775475844686ba69fa30bb6928029ce9beea88dd207b18f"}
Sep 17 17:34:20.309872 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.309851    2581 generic.go:356] "Generic (PLEG): container finished" podID="c87361b4-40c1-45cd-a813-271dfe0802d4" containerID="a7ab21c823e024a477b7cee2d9c22f8b41268b595ff9d7f7b9102d649c77ca93" exitCode=0
Sep 17 17:34:20.309951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.309883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtvtb" event={"ID":"c87361b4-40c1-45cd-a813-271dfe0802d4","Type":"ContainerDied","Data":"a7ab21c823e024a477b7cee2d9c22f8b41268b595ff9d7f7b9102d649c77ca93"}
Sep 17 17:34:20.317948 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.317913    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lg444" podStartSLOduration=3.899152372 podStartE2EDuration="21.317903055s" totalImagesPullingTime="17.418750683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:20.317872218 +0000 UTC m=+23.242344176" watchObservedRunningTime="2026-09-17 17:34:20.317903055 +0000 UTC m=+23.242375014"
Sep 17 17:34:20.318429 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.318401    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" podStartSLOduration=20.318395977 podStartE2EDuration="20.318395977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:04.284856467 +0000 UTC m=+7.209328427" watchObservedRunningTime="2026-09-17 17:34:20.318395977 +0000 UTC m=+23.242867936"
Sep 17 17:34:20.333527 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.333495    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c44np" podStartSLOduration=3.940233719 podStartE2EDuration="21.333484834s" totalImagesPullingTime="17.393251115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:20.333182193 +0000 UTC m=+23.257654151" watchObservedRunningTime="2026-09-17 17:34:20.333484834 +0000 UTC m=+23.257956793"
Sep 17 17:34:20.357490 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.357448    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-frlbs" podStartSLOduration=3.980540959 podStartE2EDuration="21.357430916s" totalImagesPullingTime="17.376889957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:20.356710273 +0000 UTC m=+23.281182232" watchObservedRunningTime="2026-09-17 17:34:20.357430916 +0000 UTC m=+23.281902875"
Sep 17 17:34:20.433108 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.433076    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9zcnt" podStartSLOduration=8.222551836 podStartE2EDuration="21.433064581s" totalImagesPullingTime="13.210512745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:20.375073628 +0000 UTC m=+23.299545599" watchObservedRunningTime="2026-09-17 17:34:20.433064581 +0000 UTC m=+23.357536541"
Sep 17 17:34:20.462152 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.459673    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mdvrr" podStartSLOduration=4.067231168 podStartE2EDuration="21.459659s" totalImagesPullingTime="17.392427832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:20.459327495 +0000 UTC m=+23.383799455" watchObservedRunningTime="2026-09-17 17:34:20.459659 +0000 UTC m=+23.384130960"
Sep 17 17:34:20.509730 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.509704    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:20.587444 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:20.587423    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:21.175135 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:21.175112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:21.175255 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:21.175229    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:21.175325 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:21.175282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:21.175384 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:21.175373    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:21.205747 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:21.205698    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:21.313569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:21.313540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6z2ss" event={"ID":"26e0af48-7c11-4be1-b8fb-19d7ee384c6e","Type":"ContainerStarted","Data":"d94fb46fd024fb24422c77d13afe826b02f51cf7529ea6ea0124d4cf53858fe9"}
Sep 17 17:34:21.315834 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:21.315806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc" event={"ID":"e3d4f629-4cd1-4cc4-b240-1ad49226294d","Type":"ContainerStarted","Data":"3bc0418ccce185bd4e7b29eb613059a468732b1c7c451672779f2eb520e11862"}
Sep 17 17:34:21.331012 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:21.330947    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6z2ss" podStartSLOduration=4.986438578 podStartE2EDuration="22.330934593s" totalImagesPullingTime="17.344496015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:21.330739168 +0000 UTC m=+24.255211127" watchObservedRunningTime="2026-09-17 17:34:21.330934593 +0000 UTC m=+24.255406552"
Sep 17 17:34:21.456252 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:21.456228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:21.523619 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:21.523581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:22.146481 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:22.146377    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:21.205717315Z","UUID":"3d247e50-75cc-4811-bcfa-b12f062361ca","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:22.148241 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:22.148210    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:22.148241 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:22.148247    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:22.174948 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:22.174921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:22.175109 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:22.175067    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:22.303359 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:22.303315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:22.404566 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:22.404515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:23.175185 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:23.175155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:23.175359 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:23.175155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:23.175359 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:23.175284    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:23.175476 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:23.175351    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:23.323941 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:23.323919    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:34:23.324435 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:23.324403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerStarted","Data":"81fce1aa4b2a57036315f89236c22438c415ed364cd0280ab1393eb71048ddd4"}
Sep 17 17:34:23.326869 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:23.326837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc" event={"ID":"e3d4f629-4cd1-4cc4-b240-1ad49226294d","Type":"ContainerStarted","Data":"5cf9dc1cc41dd31acd663a5355be6aee696dfd4de9114e17f3f95ce73ef7cc03"}
Sep 17 17:34:23.349641 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:23.349602    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz9hc" podStartSLOduration=4.67057984 podStartE2EDuration="24.349590532s" totalImagesPullingTime="19.679010692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:23.349081328 +0000 UTC m=+26.273553289" watchObservedRunningTime="2026-09-17 17:34:23.349590532 +0000 UTC m=+26.274062489"
Sep 17 17:34:23.736232 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:23.736192    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:23.836772 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:23.836722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:23.911679 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:23.911642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.078442 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:24.078374    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.175384 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:24.175357    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:24.175539 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:24.175476    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:24.737534 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:24.737338    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.839589 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:24.839564    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.932140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:24.932083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:24.932689 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:24.932664    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:25.174465 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.174447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:25.174572 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.174470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:25.174572 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:25.174525    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:25.174678 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:25.174661    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:25.333438 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.333415    2581 generic.go:356] "Generic (PLEG): container finished" podID="c87361b4-40c1-45cd-a813-271dfe0802d4" containerID="fe8d5ab486fc37109ef30200ccf528f28f09ea23b2b5684d3de63951cb7b669b" exitCode=0
Sep 17 17:34:25.333539 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.333498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtvtb" event={"ID":"c87361b4-40c1-45cd-a813-271dfe0802d4","Type":"ContainerDied","Data":"fe8d5ab486fc37109ef30200ccf528f28f09ea23b2b5684d3de63951cb7b669b"}
Sep 17 17:34:25.339834 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.339815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:34:25.340306 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.340283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerStarted","Data":"a4ced3c969bc0791a3a834d179b54cd365e56429ac0ae06f76d385a1874873b6"}
Sep 17 17:34:25.340595 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.340574    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:25.340670 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.340651    2581 scope.go:117] "RemoveContainer" containerID="1094d14007fda67497416b42053ad4f862ca20c0a08a1c84428615528fcff88e"
Sep 17 17:34:25.341083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.341064    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9zcnt"
Sep 17 17:34:25.425797 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:25.425774    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:26.174944 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.174784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:26.175496 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:26.175067    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:26.346039 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.346020    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:34:26.346399 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.346374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" event={"ID":"89ba270d-bdb8-4eb6-ae19-a569f740c1e8","Type":"ContainerStarted","Data":"ca6ae284d3db2522f93a05c3c01f3b30a104e9c0e4a8f9e437ec8e5ddde10f40"}
Sep 17 17:34:26.346496 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.346456    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:26.346757 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.346693    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:26.346841 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.346774    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:26.361165 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.361147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:26.361356 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.361340    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:26.377679 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.377638    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" podStartSLOduration=9.992518496 podStartE2EDuration="27.377621219s" totalImagesPullingTime="17.385102723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:26.377016181 +0000 UTC m=+29.301488140" watchObservedRunningTime="2026-09-17 17:34:26.377621219 +0000 UTC m=+29.302093179"
Sep 17 17:34:26.552766 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.552738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n5zs2"]
Sep 17 17:34:26.552879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.552860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:26.553065 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:26.552991    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:26.557172 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.557149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tdmfn"]
Sep 17 17:34:26.557264 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.557241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:26.557392 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:26.557371    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:26.557685 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.557668    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z485x"]
Sep 17 17:34:26.557759 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:26.557748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:26.557829 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:26.557814    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:27.353805 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:27.353775    2581 generic.go:356] "Generic (PLEG): container finished" podID="c87361b4-40c1-45cd-a813-271dfe0802d4" containerID="75f3cc71a64d4eb6d5d7397935a0c3951c6c08cafcaf703737bb46fd6d6d1a16" exitCode=0
Sep 17 17:34:27.354227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:27.353851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtvtb" event={"ID":"c87361b4-40c1-45cd-a813-271dfe0802d4","Type":"ContainerDied","Data":"75f3cc71a64d4eb6d5d7397935a0c3951c6c08cafcaf703737bb46fd6d6d1a16"}
Sep 17 17:34:27.354227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:27.353952    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:28.174612 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:28.174580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:28.174794 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:28.174674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:28.174794 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:28.174693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:28.174794 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:28.174768    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:28.174794 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:28.174676    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:28.174947 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:28.174882    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:28.356839 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:28.356820    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:28.602806 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:28.602784    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:28.760992 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:28.760952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:29.361036 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:29.361010    2581 generic.go:356] "Generic (PLEG): container finished" podID="c87361b4-40c1-45cd-a813-271dfe0802d4" containerID="d39ba42a361537eac24bdb9dd8e7d83193cb64f8296ddc685468bbe47bb2d5de" exitCode=0
Sep 17 17:34:29.361365 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:29.361104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtvtb" event={"ID":"c87361b4-40c1-45cd-a813-271dfe0802d4","Type":"ContainerDied","Data":"d39ba42a361537eac24bdb9dd8e7d83193cb64f8296ddc685468bbe47bb2d5de"}
Sep 17 17:34:29.374079 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:29.374053    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm" podUID="89ba270d-bdb8-4eb6-ae19-a569f740c1e8" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 17:34:30.174709 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:30.174676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:30.174868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:30.174759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:30.174868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:30.174858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:30.175013 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:30.174873    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:30.175013 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:30.174981    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:30.175108 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:30.175069    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:32.175400 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.175196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:32.175854 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.175251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:32.175854 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.175521    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z485x" podUID="f67faa62-0294-4089-96d5-14ecee371b01"
Sep 17 17:34:32.175854 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.175262    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:32.175854 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.175587    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5zs2" podUID="243ab975-ee79-47da-9259-405c837db145"
Sep 17 17:34:32.175854 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.175651    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tdmfn" podUID="3b795638-8c73-4961-9d78-d850444de74c"
Sep 17 17:34:32.426551 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.426467    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeReady"
Sep 17 17:34:32.426710 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.426607    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:34:32.469090 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.469049    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s2zzx"]
Sep 17 17:34:32.493118 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.493089    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h5r2d"]
Sep 17 17:34:32.493313 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.493281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.496235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.496203    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:34:32.496418 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.496397    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:34:32.496511 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.496435    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8dgbk\""
Sep 17 17:34:32.509539 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.509513    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s2zzx"]
Sep 17 17:34:32.509647 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.509558    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h5r2d"]
Sep 17 17:34:32.509647 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.509621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h5r2d"
Sep 17 17:34:32.515560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.513344    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:34:32.515560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.513812    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5qv6c\""
Sep 17 17:34:32.515560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.514001    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:34:32.515560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.514142    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:34:32.622055 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.622030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptrt5\" (UniqueName: \"kubernetes.io/projected/1dc2dde7-c0ad-4ce1-995e-a94870a19542-kube-api-access-ptrt5\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.622197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.622068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc2dde7-c0ad-4ce1-995e-a94870a19542-tmp-dir\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.622197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.622096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc2dde7-c0ad-4ce1-995e-a94870a19542-config-volume\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.622292 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.622202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv8gk\" (UniqueName: \"kubernetes.io/projected/2fdaa930-c54c-49fe-ba86-664069237f41-kube-api-access-qv8gk\") pod \"ingress-canary-h5r2d\" (UID: \"2fdaa930-c54c-49fe-ba86-664069237f41\") " pod="openshift-ingress-canary/ingress-canary-h5r2d"
Sep 17 17:34:32.622292 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.622248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.622292 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.622273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fdaa930-c54c-49fe-ba86-664069237f41-cert\") pod \"ingress-canary-h5r2d\" (UID: \"2fdaa930-c54c-49fe-ba86-664069237f41\") " pod="openshift-ingress-canary/ingress-canary-h5r2d"
Sep 17 17:34:32.639771 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.639742    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:32.717982 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.717934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:32.722932 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.722909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc2dde7-c0ad-4ce1-995e-a94870a19542-config-volume\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.723093 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.722958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv8gk\" (UniqueName: \"kubernetes.io/projected/2fdaa930-c54c-49fe-ba86-664069237f41-kube-api-access-qv8gk\") pod \"ingress-canary-h5r2d\" (UID: \"2fdaa930-c54c-49fe-ba86-664069237f41\") " pod="openshift-ingress-canary/ingress-canary-h5r2d"
Sep 17 17:34:32.723093 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.723008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.723093 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.723027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fdaa930-c54c-49fe-ba86-664069237f41-cert\") pod \"ingress-canary-h5r2d\" (UID: \"2fdaa930-c54c-49fe-ba86-664069237f41\") " pod="openshift-ingress-canary/ingress-canary-h5r2d"
Sep 17 17:34:32.723254 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.723105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptrt5\" (UniqueName: \"kubernetes.io/projected/1dc2dde7-c0ad-4ce1-995e-a94870a19542-kube-api-access-ptrt5\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.723254 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.723138    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:32.723254 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.723152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc2dde7-c0ad-4ce1-995e-a94870a19542-tmp-dir\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.723254 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.723206    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls podName:1dc2dde7-c0ad-4ce1-995e-a94870a19542 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:33.223189145 +0000 UTC m=+36.147661096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls") pod "dns-default-s2zzx" (UID: "1dc2dde7-c0ad-4ce1-995e-a94870a19542") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:32.723485 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.723464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc2dde7-c0ad-4ce1-995e-a94870a19542-tmp-dir\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.723544 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.723515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc2dde7-c0ad-4ce1-995e-a94870a19542-config-volume\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.727562 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.727537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fdaa930-c54c-49fe-ba86-664069237f41-cert\") pod \"ingress-canary-h5r2d\" (UID: \"2fdaa930-c54c-49fe-ba86-664069237f41\") " pod="openshift-ingress-canary/ingress-canary-h5r2d"
Sep 17 17:34:32.738025 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.738005    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptrt5\" (UniqueName: \"kubernetes.io/projected/1dc2dde7-c0ad-4ce1-995e-a94870a19542-kube-api-access-ptrt5\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:32.740574 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.740551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv8gk\" (UniqueName: \"kubernetes.io/projected/2fdaa930-c54c-49fe-ba86-664069237f41-kube-api-access-qv8gk\") pod \"ingress-canary-h5r2d\" (UID: \"2fdaa930-c54c-49fe-ba86-664069237f41\") " pod="openshift-ingress-canary/ingress-canary-h5r2d"
Sep 17 17:34:32.820753 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.820711    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h5r2d"
Sep 17 17:34:32.824077 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.824045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:32.824205 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.824142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:32.824263 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.824204    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:32.824263 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.824256    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:32.824361 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.824278    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs podName:3b795638-8c73-4961-9d78-d850444de74c nodeName:}" failed. No retries permitted until 2026-09-17 17:35:04.82426168 +0000 UTC m=+67.748733621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs") pod "network-metrics-daemon-tdmfn" (UID: "3b795638-8c73-4961-9d78-d850444de74c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:32.824361 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.824306    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret podName:243ab975-ee79-47da-9259-405c837db145 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:04.824289747 +0000 UTC m=+67.748761699 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret") pod "global-pull-secret-syncer-n5zs2" (UID: "243ab975-ee79-47da-9259-405c837db145") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:32.924687 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:32.924659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:32.924853 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.924836    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:32.924909 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.924859    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:32.924909 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.924873    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l4lm4 for pod openshift-network-diagnostics/network-check-target-z485x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:32.925018 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:32.924934    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4 podName:f67faa62-0294-4089-96d5-14ecee371b01 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:04.924915533 +0000 UTC m=+67.849387486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4lm4" (UniqueName: "kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4") pod "network-check-target-z485x" (UID: "f67faa62-0294-4089-96d5-14ecee371b01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:33.228243 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:33.228216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:33.228682 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:33.228372    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:33.228682 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:33.228447    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls podName:1dc2dde7-c0ad-4ce1-995e-a94870a19542 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:34.228430023 +0000 UTC m=+37.152901961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls") pod "dns-default-s2zzx" (UID: "1dc2dde7-c0ad-4ce1-995e-a94870a19542") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:33.695115 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:33.694977    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h5r2d"]
Sep 17 17:34:33.699416 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:33.699390    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdaa930_c54c_49fe_ba86_664069237f41.slice/crio-1480486ace5659622e58b2cdb38ca7cae7fe41fa622718e604551560142b9a39 WatchSource:0}: Error finding container 1480486ace5659622e58b2cdb38ca7cae7fe41fa622718e604551560142b9a39: Status 404 returned error can't find the container with id 1480486ace5659622e58b2cdb38ca7cae7fe41fa622718e604551560142b9a39
Sep 17 17:34:34.175104 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.174954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:34:34.175104 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.175007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:34:34.175104 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.175073    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:34:34.177891 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.177871    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:34.179268 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.179072    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2hx6\""
Sep 17 17:34:34.179268 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.179135    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:34.179268 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.179152    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:34.179495 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.179391    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z8blr\""
Sep 17 17:34:34.179495 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.179476    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:34.236726 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.236705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:34.237399 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:34.236794    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:34.237399 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:34.236851    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls podName:1dc2dde7-c0ad-4ce1-995e-a94870a19542 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:36.236833121 +0000 UTC m=+39.161305058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls") pod "dns-default-s2zzx" (UID: "1dc2dde7-c0ad-4ce1-995e-a94870a19542") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:34.353775 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.353560    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:34.375979 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.375944    2581 generic.go:356] "Generic (PLEG): container finished" podID="c87361b4-40c1-45cd-a813-271dfe0802d4" containerID="819b1e1fe97617387e4245be335c95f6d4767a4c1b0523582ce743ab1f3ae424" exitCode=0
Sep 17 17:34:34.376093 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.376001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtvtb" event={"ID":"c87361b4-40c1-45cd-a813-271dfe0802d4","Type":"ContainerDied","Data":"819b1e1fe97617387e4245be335c95f6d4767a4c1b0523582ce743ab1f3ae424"}
Sep 17 17:34:34.377388 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.377367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h5r2d" event={"ID":"2fdaa930-c54c-49fe-ba86-664069237f41","Type":"ContainerStarted","Data":"1480486ace5659622e58b2cdb38ca7cae7fe41fa622718e604551560142b9a39"}
Sep 17 17:34:34.422217 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:34.422189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:35.383633 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:35.383597    2581 generic.go:356] "Generic (PLEG): container finished" podID="c87361b4-40c1-45cd-a813-271dfe0802d4" containerID="a784f9f5686d851c2e7ef624930f828095856817107cbe0c1d8c159fca20f76b" exitCode=0
Sep 17 17:34:35.384175 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:35.383642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtvtb" event={"ID":"c87361b4-40c1-45cd-a813-271dfe0802d4","Type":"ContainerDied","Data":"a784f9f5686d851c2e7ef624930f828095856817107cbe0c1d8c159fca20f76b"}
Sep 17 17:34:35.583877 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:35.583838    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:35.685653 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:35.685561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:36.249948 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:36.249870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:36.250138 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:36.250032    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:36.250138 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:36.250098    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls podName:1dc2dde7-c0ad-4ce1-995e-a94870a19542 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:40.25008301 +0000 UTC m=+43.174554947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls") pod "dns-default-s2zzx" (UID: "1dc2dde7-c0ad-4ce1-995e-a94870a19542") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:36.389233 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:36.389198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtvtb" event={"ID":"c87361b4-40c1-45cd-a813-271dfe0802d4","Type":"ContainerStarted","Data":"f3fcb7e894b711b9666a48b48089e688275dcacb4053e8f34e39d260aecd262d"}
Sep 17 17:34:36.390650 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:36.390629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h5r2d" event={"ID":"2fdaa930-c54c-49fe-ba86-664069237f41","Type":"ContainerStarted","Data":"b447153eb1b27bf03392aab1e2d30f1759ffb57add627ea6e5338fdedda5a090"}
Sep 17 17:34:36.413687 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:36.413640    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vtvtb" podStartSLOduration=9.754392499 podStartE2EDuration="37.413624964s" totalImagesPullingTime="27.659232465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:36.411987686 +0000 UTC m=+39.336459639" watchObservedRunningTime="2026-09-17 17:34:36.413624964 +0000 UTC m=+39.338096922"
Sep 17 17:34:36.428366 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:36.428319    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h5r2d" podStartSLOduration=2.0816683559999998 podStartE2EDuration="4.428303389s" totalImagesPullingTime="2.346635033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:36.428135064 +0000 UTC m=+39.352607024" watchObservedRunningTime="2026-09-17 17:34:36.428303389 +0000 UTC m=+39.352775348"
Sep 17 17:34:36.544269 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:36.544243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:36.609074 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:36.609049    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:36.956529 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:36.956502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:37.055000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:37.054951    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:39.831144 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:39.831103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:39.905679 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:39.905646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:39.985072 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:39.985044    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-b47t9"]
Sep 17 17:34:39.987025 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:39.987009    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:39.992192 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:39.992158    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 17:34:39.992312 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:39.992204    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:34:39.994419 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:39.994396    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 17:34:39.994595 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:39.994579    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2fw5n\""
Sep 17 17:34:39.995915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:39.995899    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:34:40.004089 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.004068    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-b47t9"]
Sep 17 17:34:40.038877 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.038856    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 17:34:40.077444 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.077425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.077534 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.077451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.077534 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.077468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-snapshots\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.077534 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.077510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-serving-cert\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.077688 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.077565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p69nh\" (UniqueName: \"kubernetes.io/projected/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-kube-api-access-p69nh\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.077688 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.077604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-tmp\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.084751 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.084697    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"]
Sep 17 17:34:40.086493 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.086477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.088981 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.088951    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 17:34:40.089076 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.089046    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:34:40.089076 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.089066    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-54tpc\""
Sep 17 17:34:40.092163 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.090158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:34:40.092163 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.090480    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 17:34:40.108729 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.108707    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"]
Sep 17 17:34:40.178339 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.178434 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.178434 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.178434 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-snapshots\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.178563 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qghd2\" (UniqueName: \"kubernetes.io/projected/ee1b6649-932e-404d-a2e8-8960eba66ae7-kube-api-access-qghd2\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.178563 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-serving-cert\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.178657 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ee1b6649-932e-404d-a2e8-8960eba66ae7-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.178657 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p69nh\" (UniqueName: \"kubernetes.io/projected/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-kube-api-access-p69nh\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.178657 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-tmp\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.178949 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.178929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.179043 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.179009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-snapshots\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.179185 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.179170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.179610 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.179592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-tmp\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.180230 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.180214    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5"]
Sep 17 17:34:40.182031 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.182012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5"
Sep 17 17:34:40.182153 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.182130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-serving-cert\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.184483 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.184464    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mv4z8\""
Sep 17 17:34:40.184541 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.184480    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:40.184541 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.184522    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:40.191953 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.191933    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5"]
Sep 17 17:34:40.196263 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.196240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p69nh\" (UniqueName: \"kubernetes.io/projected/4575b36c-ec44-42ed-b49f-3cd2920ed4d5-kube-api-access-p69nh\") pod \"insights-operator-6f9c644dd9-b47t9\" (UID: \"4575b36c-ec44-42ed-b49f-3cd2920ed4d5\") " pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.211758 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.211739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:40.272087 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.272062    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:40.279088 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.279072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ee1b6649-932e-404d-a2e8-8960eba66ae7-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.279144 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.279120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6229\" (UniqueName: \"kubernetes.io/projected/5230ba66-378a-4c58-a52b-df47433c0377-kube-api-access-q6229\") pod \"volume-data-source-validator-5c4db8bdf9-xb2s5\" (UID: \"5230ba66-378a-4c58-a52b-df47433c0377\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5"
Sep 17 17:34:40.279177 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.279154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.279220 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.279202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qghd2\" (UniqueName: \"kubernetes.io/projected/ee1b6649-932e-404d-a2e8-8960eba66ae7-kube-api-access-qghd2\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.279294 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:40.279236    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:40.279294 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.279249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:40.279367 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:40.279301    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls podName:ee1b6649-932e-404d-a2e8-8960eba66ae7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:40.779281636 +0000 UTC m=+43.703753575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-k4shc" (UID: "ee1b6649-932e-404d-a2e8-8960eba66ae7") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:40.279367 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:40.279335    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:40.279442 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:40.279423    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls podName:1dc2dde7-c0ad-4ce1-995e-a94870a19542 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:48.27940861 +0000 UTC m=+51.203880549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls") pod "dns-default-s2zzx" (UID: "1dc2dde7-c0ad-4ce1-995e-a94870a19542") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:40.279721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.279704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ee1b6649-932e-404d-a2e8-8960eba66ae7-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.290747 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.290722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qghd2\" (UniqueName: \"kubernetes.io/projected/ee1b6649-932e-404d-a2e8-8960eba66ae7-kube-api-access-qghd2\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.296412 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.296395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-b47t9"
Sep 17 17:34:40.380503 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.380472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6229\" (UniqueName: \"kubernetes.io/projected/5230ba66-378a-4c58-a52b-df47433c0377-kube-api-access-q6229\") pod \"volume-data-source-validator-5c4db8bdf9-xb2s5\" (UID: \"5230ba66-378a-4c58-a52b-df47433c0377\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5"
Sep 17 17:34:40.393224 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.393201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6229\" (UniqueName: \"kubernetes.io/projected/5230ba66-378a-4c58-a52b-df47433c0377-kube-api-access-q6229\") pod \"volume-data-source-validator-5c4db8bdf9-xb2s5\" (UID: \"5230ba66-378a-4c58-a52b-df47433c0377\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5"
Sep 17 17:34:40.492564 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.492536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5"
Sep 17 17:34:40.577203 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.577174    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-b47t9"]
Sep 17 17:34:40.579708 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:40.579678    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4575b36c_ec44_42ed_b49f_3cd2920ed4d5.slice/crio-6c1fcffc793e47e7a9ffcb5301203c8f494cdf687eff3188a5ce0b92a16c120e WatchSource:0}: Error finding container 6c1fcffc793e47e7a9ffcb5301203c8f494cdf687eff3188a5ce0b92a16c120e: Status 404 returned error can't find the container with id 6c1fcffc793e47e7a9ffcb5301203c8f494cdf687eff3188a5ce0b92a16c120e
Sep 17 17:34:40.763743 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.763720    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5"]
Sep 17 17:34:40.766238 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:40.766215    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5230ba66_378a_4c58_a52b_df47433c0377.slice/crio-021ba1976f6db3a9482394fe97754bd338b7429af82329202d1b778138f2cd8c WatchSource:0}: Error finding container 021ba1976f6db3a9482394fe97754bd338b7429af82329202d1b778138f2cd8c: Status 404 returned error can't find the container with id 021ba1976f6db3a9482394fe97754bd338b7429af82329202d1b778138f2cd8c
Sep 17 17:34:40.783708 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:40.783686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:40.783802 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:40.783791    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:40.783849 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:40.783836    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls podName:ee1b6649-932e-404d-a2e8-8960eba66ae7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:41.783823748 +0000 UTC m=+44.708295685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-k4shc" (UID: "ee1b6649-932e-404d-a2e8-8960eba66ae7") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:41.032050 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:41.031988    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:41.404621 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:41.404524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5" event={"ID":"5230ba66-378a-4c58-a52b-df47433c0377","Type":"ContainerStarted","Data":"021ba1976f6db3a9482394fe97754bd338b7429af82329202d1b778138f2cd8c"}
Sep 17 17:34:41.406070 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:41.406034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-b47t9" event={"ID":"4575b36c-ec44-42ed-b49f-3cd2920ed4d5","Type":"ContainerStarted","Data":"6c1fcffc793e47e7a9ffcb5301203c8f494cdf687eff3188a5ce0b92a16c120e"}
Sep 17 17:34:41.792406 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:41.792365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:41.792591 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:41.792532    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:41.792645 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:41.792605    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls podName:ee1b6649-932e-404d-a2e8-8960eba66ae7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:43.792585691 +0000 UTC m=+46.717057632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-k4shc" (UID: "ee1b6649-932e-404d-a2e8-8960eba66ae7") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:42.409654 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:42.409615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5" event={"ID":"5230ba66-378a-4c58-a52b-df47433c0377","Type":"ContainerStarted","Data":"57fceb5b6aa6f1bfe6f3e1a3356aab2939c1357fb3db9dd0f40d0420060897b8"}
Sep 17 17:34:42.426988 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:42.426461    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xb2s5" podStartSLOduration=1.319484826 podStartE2EDuration="2.426442271s" totalImagesPullingTime="1.106957445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:42.425790688 +0000 UTC m=+45.350262647" watchObservedRunningTime="2026-09-17 17:34:42.426442271 +0000 UTC m=+45.350914231"
Sep 17 17:34:42.526136 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:42.526109    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:42.608912 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:42.608888    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:43.298814 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.298779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:43.385277 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.385249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:43.413874 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.413845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-b47t9" event={"ID":"4575b36c-ec44-42ed-b49f-3cd2920ed4d5","Type":"ContainerStarted","Data":"975885ddffb0da1ca3b1da670a1d07d230d8c280c4174b5594f61236a2b8a2b3"}
Sep 17 17:34:43.433310 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.433262    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-b47t9" podStartSLOduration=2.59498555 podStartE2EDuration="4.433248242s" totalImagesPullingTime="1.838262692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:43.432619442 +0000 UTC m=+46.357091401" watchObservedRunningTime="2026-09-17 17:34:43.433248242 +0000 UTC m=+46.357720201"
Sep 17 17:34:43.579982 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.579891    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nwps2"]
Sep 17 17:34:43.580380 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.580361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:43.582265 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.582250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.585034 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.585012    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:34:43.585175 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.585040    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6zzl7\""
Sep 17 17:34:43.585369 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.585344    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:34:43.612354 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.612328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nwps2"]
Sep 17 17:34:43.661393 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.661373    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:43.707604 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.707581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e71af6a9-06ca-4b00-be3d-c179616ae9ec-data-volume\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.707689 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.707639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e71af6a9-06ca-4b00-be3d-c179616ae9ec-crio-socket\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.707689 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.707656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkj4\" (UniqueName: \"kubernetes.io/projected/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-api-access-hhkj4\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.707805 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.707766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.707861 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.707801    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.808236 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.808211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e71af6a9-06ca-4b00-be3d-c179616ae9ec-data-volume\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.808319 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.808264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e71af6a9-06ca-4b00-be3d-c179616ae9ec-crio-socket\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.808319 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.808298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkj4\" (UniqueName: \"kubernetes.io/projected/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-api-access-hhkj4\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.808388 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.808352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.808388 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.808371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.808443 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.808400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:43.808443 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:43.808420    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:43.808516 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:43.808485    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:44.308471363 +0000 UTC m=+47.232943331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:43.808516 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:43.808499    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:43.808596 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:43.808516    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:43.808596 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:43.808558    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls podName:ee1b6649-932e-404d-a2e8-8960eba66ae7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.808541876 +0000 UTC m=+50.733013834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-k4shc" (UID: "ee1b6649-932e-404d-a2e8-8960eba66ae7") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:43.808596 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:43.808572    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:44.308565986 +0000 UTC m=+47.233037923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:43.808596 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.808568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e71af6a9-06ca-4b00-be3d-c179616ae9ec-data-volume\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.808740 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.808725    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e71af6a9-06ca-4b00-be3d-c179616ae9ec-crio-socket\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:43.822053 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:43.822031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkj4\" (UniqueName: \"kubernetes.io/projected/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-api-access-hhkj4\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:44.311938 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.311904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:44.311938 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.311936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:44.312165 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:44.312049    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:44.312165 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:44.312095    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:44.312165 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:44.312125    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:45.31210775 +0000 UTC m=+48.236579693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:44.312165 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:44.312145    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:45.312135968 +0000 UTC m=+48.236607905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:44.985905 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.985876    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"]
Sep 17 17:34:44.991497 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.991480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:44.994453 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.994432    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:44.995472 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.995452    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 17:34:44.995553 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.995495    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 17:34:44.995553 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.995507    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9kqdm\""
Sep 17 17:34:44.995553 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.995497    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:44.999793 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:44.999769    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"]
Sep 17 17:34:45.118497 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.118472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c894143e-e9a8-49a9-9e93-0de501d60d3b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4k49d\" (UID: \"c894143e-e9a8-49a9-9e93-0de501d60d3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.118588 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.118498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wfx\" (UniqueName: \"kubernetes.io/projected/c894143e-e9a8-49a9-9e93-0de501d60d3b-kube-api-access-g5wfx\") pod \"kube-storage-version-migrator-operator-55889f7b47-4k49d\" (UID: \"c894143e-e9a8-49a9-9e93-0de501d60d3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.118588 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.118549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c894143e-e9a8-49a9-9e93-0de501d60d3b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4k49d\" (UID: \"c894143e-e9a8-49a9-9e93-0de501d60d3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.219128 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.219103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c894143e-e9a8-49a9-9e93-0de501d60d3b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4k49d\" (UID: \"c894143e-e9a8-49a9-9e93-0de501d60d3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.219219 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.219132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wfx\" (UniqueName: \"kubernetes.io/projected/c894143e-e9a8-49a9-9e93-0de501d60d3b-kube-api-access-g5wfx\") pod \"kube-storage-version-migrator-operator-55889f7b47-4k49d\" (UID: \"c894143e-e9a8-49a9-9e93-0de501d60d3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.219219 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.219196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c894143e-e9a8-49a9-9e93-0de501d60d3b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4k49d\" (UID: \"c894143e-e9a8-49a9-9e93-0de501d60d3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.219736 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.219714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c894143e-e9a8-49a9-9e93-0de501d60d3b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4k49d\" (UID: \"c894143e-e9a8-49a9-9e93-0de501d60d3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.221394 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.221370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c894143e-e9a8-49a9-9e93-0de501d60d3b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4k49d\" (UID: \"c894143e-e9a8-49a9-9e93-0de501d60d3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.233906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.233877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wfx\" (UniqueName: \"kubernetes.io/projected/c894143e-e9a8-49a9-9e93-0de501d60d3b-kube-api-access-g5wfx\") pod \"kube-storage-version-migrator-operator-55889f7b47-4k49d\" (UID: \"c894143e-e9a8-49a9-9e93-0de501d60d3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.300430 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.300375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"
Sep 17 17:34:45.300505 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.300483    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c44np_022180cf-1016-4aa8-ad6a-9b6d20adfeba/dns-node-resolver/0.log"
Sep 17 17:34:45.319866 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.319843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:45.319979 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.319872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:45.319979 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:45.319951    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:45.320076 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:45.319985    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:45.320076 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:45.320042    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.320022299 +0000 UTC m=+50.244494238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:45.320076 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:45.320067    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.320053083 +0000 UTC m=+50.244525023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:45.568047 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:45.567947    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d"]
Sep 17 17:34:45.570595 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:45.570566    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc894143e_e9a8_49a9_9e93_0de501d60d3b.slice/crio-9c06a4f77c8927db75a9b96b8d0124704e35d07212d6c3b5592a046bbd60a040 WatchSource:0}: Error finding container 9c06a4f77c8927db75a9b96b8d0124704e35d07212d6c3b5592a046bbd60a040: Status 404 returned error can't find the container with id 9c06a4f77c8927db75a9b96b8d0124704e35d07212d6c3b5592a046bbd60a040
Sep 17 17:34:46.422575 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:46.422540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d" event={"ID":"c894143e-e9a8-49a9-9e93-0de501d60d3b","Type":"ContainerStarted","Data":"9c06a4f77c8927db75a9b96b8d0124704e35d07212d6c3b5592a046bbd60a040"}
Sep 17 17:34:46.499013 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:46.498991    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-frlbs_388ecc81-7714-4160-adbb-e4e68c6f66ef/node-ca/0.log"
Sep 17 17:34:46.995897 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:46.995859    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"]
Sep 17 17:34:47.021509 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.021474    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"]
Sep 17 17:34:47.021644 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.021616    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.024663 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.024637    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 17:34:47.025747 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.025720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 17:34:47.025747 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.025735    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:47.025904 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.025781    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2mh82\""
Sep 17 17:34:47.025904 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.025724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:47.032002 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.031979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krl2p\" (UniqueName: \"kubernetes.io/projected/9d2e9f8e-1384-4723-8f54-c35a8322d56f-kube-api-access-krl2p\") pod \"service-ca-operator-57cdc4b768-97f89\" (UID: \"9d2e9f8e-1384-4723-8f54-c35a8322d56f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.032089 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.032047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d2e9f8e-1384-4723-8f54-c35a8322d56f-config\") pod \"service-ca-operator-57cdc4b768-97f89\" (UID: \"9d2e9f8e-1384-4723-8f54-c35a8322d56f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.032195 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.032177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d2e9f8e-1384-4723-8f54-c35a8322d56f-serving-cert\") pod \"service-ca-operator-57cdc4b768-97f89\" (UID: \"9d2e9f8e-1384-4723-8f54-c35a8322d56f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.099287 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.099265    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h5r2d_2fdaa930-c54c-49fe-ba86-664069237f41/serve-healthcheck-canary/0.log"
Sep 17 17:34:47.132605 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.132578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krl2p\" (UniqueName: \"kubernetes.io/projected/9d2e9f8e-1384-4723-8f54-c35a8322d56f-kube-api-access-krl2p\") pod \"service-ca-operator-57cdc4b768-97f89\" (UID: \"9d2e9f8e-1384-4723-8f54-c35a8322d56f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.132739 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.132633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d2e9f8e-1384-4723-8f54-c35a8322d56f-config\") pod \"service-ca-operator-57cdc4b768-97f89\" (UID: \"9d2e9f8e-1384-4723-8f54-c35a8322d56f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.132739 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.132716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d2e9f8e-1384-4723-8f54-c35a8322d56f-serving-cert\") pod \"service-ca-operator-57cdc4b768-97f89\" (UID: \"9d2e9f8e-1384-4723-8f54-c35a8322d56f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.133290 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.133263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d2e9f8e-1384-4723-8f54-c35a8322d56f-config\") pod \"service-ca-operator-57cdc4b768-97f89\" (UID: \"9d2e9f8e-1384-4723-8f54-c35a8322d56f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.135528 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.135503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d2e9f8e-1384-4723-8f54-c35a8322d56f-serving-cert\") pod \"service-ca-operator-57cdc4b768-97f89\" (UID: \"9d2e9f8e-1384-4723-8f54-c35a8322d56f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.141661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.141642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krl2p\" (UniqueName: \"kubernetes.io/projected/9d2e9f8e-1384-4723-8f54-c35a8322d56f-kube-api-access-krl2p\") pod \"service-ca-operator-57cdc4b768-97f89\" (UID: \"9d2e9f8e-1384-4723-8f54-c35a8322d56f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.332663 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.332587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"
Sep 17 17:34:47.333638 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.333609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:47.333782 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.333649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:47.333782 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:47.333744    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:47.333782 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:47.333775    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:47.333912 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:47.333829    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.333810713 +0000 UTC m=+54.258282668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:47.333912 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:47.333848    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.33383908 +0000 UTC m=+54.258311022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:47.334814 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.334791    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr"]
Sep 17 17:34:47.359937 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.359920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr"
Sep 17 17:34:47.363139 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.363112    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr"]
Sep 17 17:34:47.363399 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.363377    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-j64g6\""
Sep 17 17:34:47.434560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.434534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xkp\" (UniqueName: \"kubernetes.io/projected/c718a53d-0052-45aa-ac2a-42e44ab33439-kube-api-access-f2xkp\") pod \"network-check-source-7cbc45fddc-cq2fr\" (UID: \"c718a53d-0052-45aa-ac2a-42e44ab33439\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr"
Sep 17 17:34:47.535318 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.535284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xkp\" (UniqueName: \"kubernetes.io/projected/c718a53d-0052-45aa-ac2a-42e44ab33439-kube-api-access-f2xkp\") pod \"network-check-source-7cbc45fddc-cq2fr\" (UID: \"c718a53d-0052-45aa-ac2a-42e44ab33439\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr"
Sep 17 17:34:47.545272 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.545239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xkp\" (UniqueName: \"kubernetes.io/projected/c718a53d-0052-45aa-ac2a-42e44ab33439-kube-api-access-f2xkp\") pod \"network-check-source-7cbc45fddc-cq2fr\" (UID: \"c718a53d-0052-45aa-ac2a-42e44ab33439\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr"
Sep 17 17:34:47.671233 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.671201    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr"
Sep 17 17:34:47.838036 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.838006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:47.838178 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:47.838121    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:47.838178 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:47.838177    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls podName:ee1b6649-932e-404d-a2e8-8960eba66ae7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:55.838164133 +0000 UTC m=+58.762636070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-k4shc" (UID: "ee1b6649-932e-404d-a2e8-8960eba66ae7") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:47.898364 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.898305    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89"]
Sep 17 17:34:47.950057 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:47.950015    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr"]
Sep 17 17:34:48.004730 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:48.004703    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2e9f8e_1384_4723_8f54_c35a8322d56f.slice/crio-1929aa8b45324ba59491fe69431cbe4a7c183aa6ed3c401b9c825c3e98adbb01 WatchSource:0}: Error finding container 1929aa8b45324ba59491fe69431cbe4a7c183aa6ed3c401b9c825c3e98adbb01: Status 404 returned error can't find the container with id 1929aa8b45324ba59491fe69431cbe4a7c183aa6ed3c401b9c825c3e98adbb01
Sep 17 17:34:48.005208 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:48.005184    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc718a53d_0052_45aa_ac2a_42e44ab33439.slice/crio-62dd52840d718417f03abf3f13606d725ef97a2acafe6273ad02e923a08f022a WatchSource:0}: Error finding container 62dd52840d718417f03abf3f13606d725ef97a2acafe6273ad02e923a08f022a: Status 404 returned error can't find the container with id 62dd52840d718417f03abf3f13606d725ef97a2acafe6273ad02e923a08f022a
Sep 17 17:34:48.343926 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:48.343898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:34:48.344083 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:48.344065    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:48.344133 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:48.344123    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls podName:1dc2dde7-c0ad-4ce1-995e-a94870a19542 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:04.344109193 +0000 UTC m=+67.268581130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls") pod "dns-default-s2zzx" (UID: "1dc2dde7-c0ad-4ce1-995e-a94870a19542") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:48.429604 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:48.429579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89" event={"ID":"9d2e9f8e-1384-4723-8f54-c35a8322d56f","Type":"ContainerStarted","Data":"1929aa8b45324ba59491fe69431cbe4a7c183aa6ed3c401b9c825c3e98adbb01"}
Sep 17 17:34:48.431144 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:48.431120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d" event={"ID":"c894143e-e9a8-49a9-9e93-0de501d60d3b","Type":"ContainerStarted","Data":"ca7926fe543b39928d84730ccbb76b14d4be4783966ef47ec6b19883fd7d7336"}
Sep 17 17:34:48.432263 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:48.432236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr" event={"ID":"c718a53d-0052-45aa-ac2a-42e44ab33439","Type":"ContainerStarted","Data":"62dd52840d718417f03abf3f13606d725ef97a2acafe6273ad02e923a08f022a"}
Sep 17 17:34:48.449604 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:48.449566    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d" podStartSLOduration=1.9841988289999999 podStartE2EDuration="4.449552334s" totalImagesPullingTime="2.465353505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:48.448710941 +0000 UTC m=+51.373182899" watchObservedRunningTime="2026-09-17 17:34:48.449552334 +0000 UTC m=+51.374024292"
Sep 17 17:34:50.101391 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.101359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:50.137127 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.137100    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2"]
Sep 17 17:34:50.152066 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.152040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2"
Sep 17 17:34:50.155987 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.154952    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d5glp\""
Sep 17 17:34:50.155987 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.155614    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 17:34:50.155987 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.154777    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:50.157723 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.157632    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2"]
Sep 17 17:34:50.158097 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.158061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwvbc\" (UniqueName: \"kubernetes.io/projected/cd8188a2-f9f7-405a-a26d-40551aa263df-kube-api-access-mwvbc\") pod \"migrator-76794cbd6c-7cfs2\" (UID: \"cd8188a2-f9f7-405a-a26d-40551aa263df\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2"
Sep 17 17:34:50.196257 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.196231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:50.258461 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.258434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwvbc\" (UniqueName: \"kubernetes.io/projected/cd8188a2-f9f7-405a-a26d-40551aa263df-kube-api-access-mwvbc\") pod \"migrator-76794cbd6c-7cfs2\" (UID: \"cd8188a2-f9f7-405a-a26d-40551aa263df\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2"
Sep 17 17:34:50.272565 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.272544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwvbc\" (UniqueName: \"kubernetes.io/projected/cd8188a2-f9f7-405a-a26d-40551aa263df-kube-api-access-mwvbc\") pod \"migrator-76794cbd6c-7cfs2\" (UID: \"cd8188a2-f9f7-405a-a26d-40551aa263df\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2"
Sep 17 17:34:50.300129 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.300089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:50.463943 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:50.463917    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2"
Sep 17 17:34:51.367767 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:51.367744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:51.368135 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:51.367854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:51.368135 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:51.367858    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:51.368135 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:51.367917    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:51.368135 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:51.368052    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:59.368027554 +0000 UTC m=+62.292499505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:51.368135 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:51.368074    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:34:59.368064877 +0000 UTC m=+62.292536814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:51.613061 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:51.613033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2"]
Sep 17 17:34:51.614817 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:34:51.614794    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8188a2_f9f7_405a_a26d_40551aa263df.slice/crio-336c286769bcf73b6d2fa024f4da65b5c36d1d9b84cfe3d2b5acec6c507b6d83 WatchSource:0}: Error finding container 336c286769bcf73b6d2fa024f4da65b5c36d1d9b84cfe3d2b5acec6c507b6d83: Status 404 returned error can't find the container with id 336c286769bcf73b6d2fa024f4da65b5c36d1d9b84cfe3d2b5acec6c507b6d83
Sep 17 17:34:52.315405 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:52.315369    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:52.382818 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:52.382788    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:52.445134 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:52.445110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89" event={"ID":"9d2e9f8e-1384-4723-8f54-c35a8322d56f","Type":"ContainerStarted","Data":"cc7eab7aab1cce1cde0c539e68fe96325631220fb811bcd18b225ebeba8647bc"}
Sep 17 17:34:52.446367 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:52.446344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2" event={"ID":"cd8188a2-f9f7-405a-a26d-40551aa263df","Type":"ContainerStarted","Data":"336c286769bcf73b6d2fa024f4da65b5c36d1d9b84cfe3d2b5acec6c507b6d83"}
Sep 17 17:34:52.447865 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:52.447844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr" event={"ID":"c718a53d-0052-45aa-ac2a-42e44ab33439","Type":"ContainerStarted","Data":"d722ae4d96bcdafea43fe8c7dc76936d0d5d24e57dcf8f497f170e7610a57429"}
Sep 17 17:34:52.462483 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:52.462429    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89" podStartSLOduration=3.144733779 podStartE2EDuration="6.462414462s" totalImagesPullingTime="3.317680683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:52.462285246 +0000 UTC m=+55.386757206" watchObservedRunningTime="2026-09-17 17:34:52.462414462 +0000 UTC m=+55.386886422"
Sep 17 17:34:52.488132 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:52.487987    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-cq2fr" podStartSLOduration=2.160454459 podStartE2EDuration="5.487956746s" totalImagesPullingTime="3.327502287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:52.487933116 +0000 UTC m=+55.412405079" watchObservedRunningTime="2026-09-17 17:34:52.487956746 +0000 UTC m=+55.412428706"
Sep 17 17:34:52.749153 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:52.749123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:52.852073 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:52.852042    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:53.022192 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:53.022123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:53.126674 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:53.126645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:53.451431 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:53.451395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2" event={"ID":"cd8188a2-f9f7-405a-a26d-40551aa263df","Type":"ContainerStarted","Data":"d74da10595d5658a62fc72bec599032a55ab8a70b2ccb1b36b776cab044aa7fc"}
Sep 17 17:34:54.456352 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:54.456315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2" event={"ID":"cd8188a2-f9f7-405a-a26d-40551aa263df","Type":"ContainerStarted","Data":"a6782dd9e696343aec31af09e701b93e32472c5726eda33c696591a59e14145e"}
Sep 17 17:34:54.478484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:54.478433    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7cfs2" podStartSLOduration=2.7758155110000002 podStartE2EDuration="4.478414752s" totalImagesPullingTime="1.702599241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:54.477739219 +0000 UTC m=+57.402211179" watchObservedRunningTime="2026-09-17 17:34:54.478414752 +0000 UTC m=+57.402886712"
Sep 17 17:34:54.573249 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:54.573217    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:54.672024 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:54.671996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:54.733629 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:54.733575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:55.859801 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:55.859768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:55.898773 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:55.898744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:34:55.898904 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:55.898885    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:55.898956 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:55.898949    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls podName:ee1b6649-932e-404d-a2e8-8960eba66ae7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.898933569 +0000 UTC m=+74.823405512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-k4shc" (UID: "ee1b6649-932e-404d-a2e8-8960eba66ae7") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:56.723633 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:56.723604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:58.769348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:58.769303    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:58.870012 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:58.869982    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:58.970144 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:58.970112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:59.069635 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:59.069568    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:59.170469 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:59.170447    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:59.370604 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:59.370549    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbjdm"
Sep 17 17:34:59.423766 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:59.423734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:59.423890 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:59.423849    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:59.423958 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:34:59.423893    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls podName:e71af6a9-06ca-4b00-be3d-c179616ae9ec nodeName:}" failed. No retries permitted until 2026-09-17 17:35:15.42387965 +0000 UTC m=+78.348351587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwps2" (UID: "e71af6a9-06ca-4b00-be3d-c179616ae9ec") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:59.424068 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:59.424052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:59.424484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:59.424465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e71af6a9-06ca-4b00-be3d-c179616ae9ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:34:59.668431 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:34:59.668365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.475089 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:00.475047    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.677467 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:00.677438    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.871902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:00.871837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:01.066530 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:01.066499    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.227057 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:02.227028    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.287853 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:02.287818    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.463191 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:02.463165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.527649 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:02.527580    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.661547 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:02.661519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.880712 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:02.880656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:03.057413 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:03.057388    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:03.126798 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:03.126775    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:03.857747 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:03.857719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.263210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.263179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.363168 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.363138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:35:04.364608 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.364586    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.365773 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.365747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dc2dde7-c0ad-4ce1-995e-a94870a19542-metrics-tls\") pod \"dns-default-s2zzx\" (UID: \"1dc2dde7-c0ad-4ce1-995e-a94870a19542\") " pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:35:04.463919 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.463889    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.607507 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.607446    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8dgbk\""
Sep 17 17:35:04.615698 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.615678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:35:04.657529 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.657496    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.867523 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.867460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:35:04.867523 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.867498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:35:04.870348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.870319    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:04.870487 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.870390    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:04.878494 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:35:04.878467    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:35:04.878601 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:35:04.878523    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs podName:3b795638-8c73-4961-9d78-d850444de74c nodeName:}" failed. No retries permitted until 2026-09-17 17:36:08.878509183 +0000 UTC m=+131.802981119 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs") pod "network-metrics-daemon-tdmfn" (UID: "3b795638-8c73-4961-9d78-d850444de74c") : secret "metrics-daemon-secret" not found
Sep 17 17:35:04.880484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.880461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/243ab975-ee79-47da-9259-405c837db145-original-pull-secret\") pod \"global-pull-secret-syncer-n5zs2\" (UID: \"243ab975-ee79-47da-9259-405c837db145\") " pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:35:04.889337 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.889313    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s2zzx"]
Sep 17 17:35:04.893351 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:04.893324    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc2dde7_c0ad_4ce1_995e_a94870a19542.slice/crio-865f792b3c148bab5db64b3b25cda9257db114a3dd22044c3b670bfb9eb7f13d WatchSource:0}: Error finding container 865f792b3c148bab5db64b3b25cda9257db114a3dd22044c3b670bfb9eb7f13d: Status 404 returned error can't find the container with id 865f792b3c148bab5db64b3b25cda9257db114a3dd22044c3b670bfb9eb7f13d
Sep 17 17:35:04.953090 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.953065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.968128 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.968108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:35:04.970203 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:04.970188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lm4\" (UniqueName: \"kubernetes.io/projected/f67faa62-0294-4089-96d5-14ecee371b01-kube-api-access-l4lm4\") pod \"network-check-target-z485x\" (UID: \"f67faa62-0294-4089-96d5-14ecee371b01\") " pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:35:05.060434 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.060412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:05.086491 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.086474    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5zs2"
Sep 17 17:35:05.102514 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.102497    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z8blr\""
Sep 17 17:35:05.110664 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.110646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:35:05.272186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.272162    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:05.357745 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.355733    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n5zs2"]
Sep 17 17:35:05.361328 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:05.361298    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243ab975_ee79_47da_9259_405c837db145.slice/crio-45404d330d014e0d112c626801fc69f87599b27418f5b11f174665031fa4b2ab WatchSource:0}: Error finding container 45404d330d014e0d112c626801fc69f87599b27418f5b11f174665031fa4b2ab: Status 404 returned error can't find the container with id 45404d330d014e0d112c626801fc69f87599b27418f5b11f174665031fa4b2ab
Sep 17 17:35:05.394679 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.394609    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z485x"]
Sep 17 17:35:05.399639 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:05.399610    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67faa62_0294_4089_96d5_14ecee371b01.slice/crio-966ce2f41cce24ac29eb97f7a4f8ede9bf3d4da854da71d35d8feb36945d7cd2 WatchSource:0}: Error finding container 966ce2f41cce24ac29eb97f7a4f8ede9bf3d4da854da71d35d8feb36945d7cd2: Status 404 returned error can't find the container with id 966ce2f41cce24ac29eb97f7a4f8ede9bf3d4da854da71d35d8feb36945d7cd2
Sep 17 17:35:05.496050 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.496021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n5zs2" event={"ID":"243ab975-ee79-47da-9259-405c837db145","Type":"ContainerStarted","Data":"45404d330d014e0d112c626801fc69f87599b27418f5b11f174665031fa4b2ab"}
Sep 17 17:35:05.497482 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.497459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s2zzx" event={"ID":"1dc2dde7-c0ad-4ce1-995e-a94870a19542","Type":"ContainerStarted","Data":"865f792b3c148bab5db64b3b25cda9257db114a3dd22044c3b670bfb9eb7f13d"}
Sep 17 17:35:05.499164 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.499140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z485x" event={"ID":"f67faa62-0294-4089-96d5-14ecee371b01","Type":"ContainerStarted","Data":"fc1708e617af770121cfbff888080b62de34f38953fa616e63edaee0284ad6df"}
Sep 17 17:35:05.499274 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.499187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z485x" event={"ID":"f67faa62-0294-4089-96d5-14ecee371b01","Type":"ContainerStarted","Data":"966ce2f41cce24ac29eb97f7a4f8ede9bf3d4da854da71d35d8feb36945d7cd2"}
Sep 17 17:35:05.499341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.499275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:35:05.519638 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.519598    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z485x" podStartSLOduration=66.51958276 podStartE2EDuration="1m6.51958276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:05.519010797 +0000 UTC m=+68.443482756" watchObservedRunningTime="2026-09-17 17:35:05.51958276 +0000 UTC m=+68.444054722"
Sep 17 17:35:05.860603 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:05.860576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.461147 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:06.461111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.670903 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:06.670871    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.907577 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:06.907491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:07.254477 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:07.254445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:07.507635 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:07.507555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s2zzx" event={"ID":"1dc2dde7-c0ad-4ce1-995e-a94870a19542","Type":"ContainerStarted","Data":"742ec74214f18b66e25a1878585dfc17c3f77c5cda546fd207c4c2510d501fc4"}
Sep 17 17:35:07.507635 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:07.507616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s2zzx" event={"ID":"1dc2dde7-c0ad-4ce1-995e-a94870a19542","Type":"ContainerStarted","Data":"c7146c21e88256268ca931ab1899f858ed8a7107ebbc6c48bed852a945158bea"}
Sep 17 17:35:07.508162 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:07.507707    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:35:07.526000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:07.525887    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s2zzx" podStartSLOduration=33.666852162 podStartE2EDuration="35.525874121s" totalImagesPullingTime="1.859021959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:07.524453355 +0000 UTC m=+70.448925314" watchObservedRunningTime="2026-09-17 17:35:07.525874121 +0000 UTC m=+70.450346079"
Sep 17 17:35:07.863484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:07.863408    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.059246 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:08.059203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.460533 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:08.460498    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.857674 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:08.857561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.921272 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:08.921242    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:09.265261 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:09.265231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:09.489149 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:09.489103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:09.516072 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:09.516039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n5zs2" event={"ID":"243ab975-ee79-47da-9259-405c837db145","Type":"ContainerStarted","Data":"66e667d98706de04f032bc00ec1e632e2c92955b8fe6ff9321e63e7708a811ab"}
Sep 17 17:35:09.535905 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:09.535861    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n5zs2" podStartSLOduration=66.756385342 podStartE2EDuration="1m10.535846423s" totalImagesPullingTime="3.779461081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:09.535545791 +0000 UTC m=+72.460017751" watchObservedRunningTime="2026-09-17 17:35:09.535846423 +0000 UTC m=+72.460318377"
Sep 17 17:35:11.919825 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:11.919685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:35:11.922367 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:11.922336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee1b6649-932e-404d-a2e8-8960eba66ae7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-k4shc\" (UID: \"ee1b6649-932e-404d-a2e8-8960eba66ae7\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:35:12.198457 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:12.198432    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-54tpc\""
Sep 17 17:35:12.206470 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:12.206452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"
Sep 17 17:35:12.487655 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:12.487594    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc"]
Sep 17 17:35:12.490811 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:12.490772    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1b6649_932e_404d_a2e8_8960eba66ae7.slice/crio-330b4c851d2241d1d5141c6b5e62ebc5862e76b01ce2b46a4e998d137fa7c100 WatchSource:0}: Error finding container 330b4c851d2241d1d5141c6b5e62ebc5862e76b01ce2b46a4e998d137fa7c100: Status 404 returned error can't find the container with id 330b4c851d2241d1d5141c6b5e62ebc5862e76b01ce2b46a4e998d137fa7c100
Sep 17 17:35:12.528066 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:12.528037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc" event={"ID":"ee1b6649-932e-404d-a2e8-8960eba66ae7","Type":"ContainerStarted","Data":"330b4c851d2241d1d5141c6b5e62ebc5862e76b01ce2b46a4e998d137fa7c100"}
Sep 17 17:35:14.463390 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:14.463357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:14.537112 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:14.537084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc" event={"ID":"ee1b6649-932e-404d-a2e8-8960eba66ae7","Type":"ContainerStarted","Data":"b04adb0177fe6f2db213427f367ad7229cb02f469e0fe857e5fdef77c3091461"}
Sep 17 17:35:14.556039 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:14.556001    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-k4shc" podStartSLOduration=32.927738182 podStartE2EDuration="34.555986556s" totalImagesPullingTime="1.628248374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:14.555408661 +0000 UTC m=+77.479880622" watchObservedRunningTime="2026-09-17 17:35:14.555986556 +0000 UTC m=+77.480458512"
Sep 17 17:35:14.560902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:14.560877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:14.653882 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:14.653859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:14.784558 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:14.784503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:15.442785 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:15.442742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:35:15.445362 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:15.445330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e71af6a9-06ca-4b00-be3d-c179616ae9ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwps2\" (UID: \"e71af6a9-06ca-4b00-be3d-c179616ae9ec\") " pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:35:15.693297 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:15.693237    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6zzl7\""
Sep 17 17:35:15.701395 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:15.701372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nwps2"
Sep 17 17:35:15.975118 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:15.975089    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nwps2"]
Sep 17 17:35:15.978005 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:15.977953    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71af6a9_06ca_4b00_be3d_c179616ae9ec.slice/crio-c313755f0d587205aefed368113d7550934793c8999628948536ff85f5dad585 WatchSource:0}: Error finding container c313755f0d587205aefed368113d7550934793c8999628948536ff85f5dad585: Status 404 returned error can't find the container with id c313755f0d587205aefed368113d7550934793c8999628948536ff85f5dad585
Sep 17 17:35:16.546364 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:16.546324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwps2" event={"ID":"e71af6a9-06ca-4b00-be3d-c179616ae9ec","Type":"ContainerStarted","Data":"d95d93979c01a8957af118e881249d363ecdd56ce65a07a65db2d7f2564242f7"}
Sep 17 17:35:16.546510 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:16.546390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwps2" event={"ID":"e71af6a9-06ca-4b00-be3d-c179616ae9ec","Type":"ContainerStarted","Data":"c313755f0d587205aefed368113d7550934793c8999628948536ff85f5dad585"}
Sep 17 17:35:17.513268 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:17.513244    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s2zzx"
Sep 17 17:35:17.551923 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:17.551891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwps2" event={"ID":"e71af6a9-06ca-4b00-be3d-c179616ae9ec","Type":"ContainerStarted","Data":"cb7174fd697f190649b9eefcc69a9236e391143db3d222d852a3cb9c6ff0c321"}
Sep 17 17:35:18.510673 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.510640    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"]
Sep 17 17:35:18.512650 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.512632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"
Sep 17 17:35:18.517000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.516977    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 17:35:18.517412 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.516979    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:35:18.517412 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.516982    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:35:18.517412 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.516981    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bvxw5\""
Sep 17 17:35:18.517412 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.517011    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:35:18.535622 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.535601    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"]
Sep 17 17:35:18.557400 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.557378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwps2" event={"ID":"e71af6a9-06ca-4b00-be3d-c179616ae9ec","Type":"ContainerStarted","Data":"b0afc98fcff4ed77d13848030d58e9f4d3b8e83cf6504ddad6f27d406ffdeb16"}
Sep 17 17:35:18.565383 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.565360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed796e51-42f8-44b2-b3bf-2d662d4b4cfb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c7986dd5d-qxnkd\" (UID: \"ed796e51-42f8-44b2-b3bf-2d662d4b4cfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"
Sep 17 17:35:18.565484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.565410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntp72\" (UniqueName: \"kubernetes.io/projected/ed796e51-42f8-44b2-b3bf-2d662d4b4cfb-kube-api-access-ntp72\") pod \"managed-serviceaccount-addon-agent-c7986dd5d-qxnkd\" (UID: \"ed796e51-42f8-44b2-b3bf-2d662d4b4cfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"
Sep 17 17:35:18.599135 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.599099    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nwps2" podStartSLOduration=33.813021809 podStartE2EDuration="35.59908613s" totalImagesPullingTime="1.786064321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:18.598633716 +0000 UTC m=+81.523105676" watchObservedRunningTime="2026-09-17 17:35:18.59908613 +0000 UTC m=+81.523558085"
Sep 17 17:35:18.666571 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.666544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntp72\" (UniqueName: \"kubernetes.io/projected/ed796e51-42f8-44b2-b3bf-2d662d4b4cfb-kube-api-access-ntp72\") pod \"managed-serviceaccount-addon-agent-c7986dd5d-qxnkd\" (UID: \"ed796e51-42f8-44b2-b3bf-2d662d4b4cfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"
Sep 17 17:35:18.666665 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.666599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed796e51-42f8-44b2-b3bf-2d662d4b4cfb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c7986dd5d-qxnkd\" (UID: \"ed796e51-42f8-44b2-b3bf-2d662d4b4cfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"
Sep 17 17:35:18.668926 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.668905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed796e51-42f8-44b2-b3bf-2d662d4b4cfb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c7986dd5d-qxnkd\" (UID: \"ed796e51-42f8-44b2-b3bf-2d662d4b4cfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"
Sep 17 17:35:18.698461 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.698436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntp72\" (UniqueName: \"kubernetes.io/projected/ed796e51-42f8-44b2-b3bf-2d662d4b4cfb-kube-api-access-ntp72\") pod \"managed-serviceaccount-addon-agent-c7986dd5d-qxnkd\" (UID: \"ed796e51-42f8-44b2-b3bf-2d662d4b4cfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"
Sep 17 17:35:18.771211 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.771143    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:18.838198 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.838177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"
Sep 17 17:35:18.874346 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.874317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:18.971869 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:18.971840    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:19.065922 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:19.065858    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:19.115196 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:19.115174    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd"]
Sep 17 17:35:19.118307 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:19.118287    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded796e51_42f8_44b2_b3bf_2d662d4b4cfb.slice/crio-a21cd80102a7d3b3c7f13f0f3978d6b7d61d77b0165e6bac9ac278f0ec753c89 WatchSource:0}: Error finding container a21cd80102a7d3b3c7f13f0f3978d6b7d61d77b0165e6bac9ac278f0ec753c89: Status 404 returned error can't find the container with id a21cd80102a7d3b3c7f13f0f3978d6b7d61d77b0165e6bac9ac278f0ec753c89
Sep 17 17:35:19.363745 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:19.363672    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:19.473567 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:19.473470    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:19.562855 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:19.562816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd" event={"ID":"ed796e51-42f8-44b2-b3bf-2d662d4b4cfb","Type":"ContainerStarted","Data":"a21cd80102a7d3b3c7f13f0f3978d6b7d61d77b0165e6bac9ac278f0ec753c89"}
Sep 17 17:35:21.722476 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.722448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:21.761599 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.761575    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"]
Sep 17 17:35:21.764663 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.764647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.767225 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.767206    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 17:35:21.768392 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.768364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:35:21.768497 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.768369    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 17:35:21.768561 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.768496    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gtppd\""
Sep 17 17:35:21.781484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.781463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"]
Sep 17 17:35:21.802072 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.802051    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:21.889105 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.889080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.889200 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.889116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzw8s\" (UniqueName: \"kubernetes.io/projected/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-kube-api-access-tzw8s\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.889200 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.889143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.889282 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.889209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.952988 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.952950    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:21.990015 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.989990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.990100 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.990020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzw8s\" (UniqueName: \"kubernetes.io/projected/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-kube-api-access-tzw8s\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.990145 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.990133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.990180 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.990168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.990783 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.990760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.992371 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.992350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.992446 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.992352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:21.998691 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:21.998666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzw8s\" (UniqueName: \"kubernetes.io/projected/cd28df02-e0f6-401d-ab7f-61f50bc4ed39-kube-api-access-tzw8s\") pod \"prometheus-operator-7bd5d56d6-l466x\" (UID: \"cd28df02-e0f6-401d-ab7f-61f50bc4ed39\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:22.073790 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:22.073769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"
Sep 17 17:35:22.346652 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:22.346626    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-l466x"]
Sep 17 17:35:22.348593 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:22.348564    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd28df02_e0f6_401d_ab7f_61f50bc4ed39.slice/crio-d58615f2498216d2deecf83089a44c94189d7ef49060fb5cb4f5dbcf93f03c83 WatchSource:0}: Error finding container d58615f2498216d2deecf83089a44c94189d7ef49060fb5cb4f5dbcf93f03c83: Status 404 returned error can't find the container with id d58615f2498216d2deecf83089a44c94189d7ef49060fb5cb4f5dbcf93f03c83
Sep 17 17:35:22.575398 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:22.575362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd" event={"ID":"ed796e51-42f8-44b2-b3bf-2d662d4b4cfb","Type":"ContainerStarted","Data":"2ca7e6e3586c01369624c340897fbc38ee2e5e1b1827cdc9d6be9bf00a32b19e"}
Sep 17 17:35:22.576660 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:22.576639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x" event={"ID":"cd28df02-e0f6-401d-ab7f-61f50bc4ed39","Type":"ContainerStarted","Data":"d58615f2498216d2deecf83089a44c94189d7ef49060fb5cb4f5dbcf93f03c83"}
Sep 17 17:35:22.596461 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:22.596394    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c7986dd5d-qxnkd" podStartSLOduration=2.073590854 podStartE2EDuration="4.596380204s" totalImagesPullingTime="2.52278935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:22.595145123 +0000 UTC m=+85.519617096" watchObservedRunningTime="2026-09-17 17:35:22.596380204 +0000 UTC m=+85.520852162"
Sep 17 17:35:24.010232 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:24.010153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:24.589945 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:24.589912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x" event={"ID":"cd28df02-e0f6-401d-ab7f-61f50bc4ed39","Type":"ContainerStarted","Data":"951df121d0fa9e39d5c9250daa0c1bf31be995c1e7b80fc03bb231a3d33fffe8"}
Sep 17 17:35:24.590134 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:24.590007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x" event={"ID":"cd28df02-e0f6-401d-ab7f-61f50bc4ed39","Type":"ContainerStarted","Data":"b23991e5e5ced4c68ff0ef0d84b91b95e7484019b575be788d975130bab0ce71"}
Sep 17 17:35:24.608039 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:24.607988    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-l466x" podStartSLOduration=2.2007437579999998 podStartE2EDuration="3.607955017s" totalImagesPullingTime="1.407211259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:24.607684364 +0000 UTC m=+87.532156322" watchObservedRunningTime="2026-09-17 17:35:24.607955017 +0000 UTC m=+87.532426975"
Sep 17 17:35:25.873095 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:25.873065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:26.103317 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.103286    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:26.140741 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.140673    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"]
Sep 17 17:35:26.167665 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.167641    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-qpqth"]
Sep 17 17:35:26.167777 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.167754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.170752 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.170697    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cgx5f\""
Sep 17 17:35:26.170752 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.170741    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:35:26.170928 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.170805    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 17:35:26.181431 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.181412    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"]
Sep 17 17:35:26.181522 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.181471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-qpqth"]
Sep 17 17:35:26.181522 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.181492    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xshpf"]
Sep 17 17:35:26.181734 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.181715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.184383 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.184361    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:35:26.184500 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.184471    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ncfjz\""
Sep 17 17:35:26.184593 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.184514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:35:26.184593 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.184573    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:35:26.201567 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.201550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.204155 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.204137    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:35:26.204155 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.204162    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:35:26.204332 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.204213    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:35:26.204407 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.204336    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-twsbr\""
Sep 17 17:35:26.205669 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.205649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:26.306580 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.306557    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:26.317485 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08297acf-4eb9-44de-9039-d828bc27d430-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.317560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lckkb\" (UniqueName: \"kubernetes.io/projected/43182a8d-c31e-4b95-94e5-9e94ca06d6af-kube-api-access-lckkb\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.317560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43182a8d-c31e-4b95-94e5-9e94ca06d6af-metrics-client-ca\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.317560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf7jx\" (UniqueName: \"kubernetes.io/projected/08297acf-4eb9-44de-9039-d828bc27d430-kube-api-access-mf7jx\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.317560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.317721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08297acf-4eb9-44de-9039-d828bc27d430-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.317721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-accelerators-collector-config\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.317721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43182a8d-c31e-4b95-94e5-9e94ca06d6af-sys\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.317721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.317721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43182a8d-c31e-4b95-94e5-9e94ca06d6af-root\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.317721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-wtmp\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.317940 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.317940 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg7bf\" (UniqueName: \"kubernetes.io/projected/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-api-access-kg7bf\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.317940 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08297acf-4eb9-44de-9039-d828bc27d430-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.317940 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.317940 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.317940 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.317940 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-tls\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.317940 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.317927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-textfile\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.412340 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.412270    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:26.418608 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43182a8d-c31e-4b95-94e5-9e94ca06d6af-sys\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.418717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.418717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43182a8d-c31e-4b95-94e5-9e94ca06d6af-root\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.418717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-wtmp\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.418717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43182a8d-c31e-4b95-94e5-9e94ca06d6af-sys\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.418881 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43182a8d-c31e-4b95-94e5-9e94ca06d6af-root\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.418881 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-wtmp\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.418881 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.418881 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg7bf\" (UniqueName: \"kubernetes.io/projected/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-api-access-kg7bf\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.418881 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08297acf-4eb9-44de-9039-d828bc27d430-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.419150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.419150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.419150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.419150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.418978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-tls\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.419150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.419016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-textfile\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.419150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.419045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08297acf-4eb9-44de-9039-d828bc27d430-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.419150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.419069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lckkb\" (UniqueName: \"kubernetes.io/projected/43182a8d-c31e-4b95-94e5-9e94ca06d6af-kube-api-access-lckkb\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.419470 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.419249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.419470 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:35:26.419332    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 17:35:26.419470 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:35:26.419392    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-tls podName:43182a8d-c31e-4b95-94e5-9e94ca06d6af nodeName:}" failed. No retries permitted until 2026-09-17 17:35:26.919369318 +0000 UTC m=+89.843841271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-tls") pod "node-exporter-xshpf" (UID: "43182a8d-c31e-4b95-94e5-9e94ca06d6af") : secret "node-exporter-tls" not found
Sep 17 17:35:26.419470 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.419394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-textfile\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.419470 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.419430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43182a8d-c31e-4b95-94e5-9e94ca06d6af-metrics-client-ca\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.419719 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.419480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf7jx\" (UniqueName: \"kubernetes.io/projected/08297acf-4eb9-44de-9039-d828bc27d430-kube-api-access-mf7jx\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.419719 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.419509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.419719 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.419660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.420189 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.420167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43182a8d-c31e-4b95-94e5-9e94ca06d6af-metrics-client-ca\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.420319 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.420296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.420421 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.420400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08297acf-4eb9-44de-9039-d828bc27d430-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.421183 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.421156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-accelerators-collector-config\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.421183 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.421080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08297acf-4eb9-44de-9039-d828bc27d430-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.421332 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.421267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.421634 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.421609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08297acf-4eb9-44de-9039-d828bc27d430-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.421722 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.421688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-accelerators-collector-config\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.421886 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.421868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08297acf-4eb9-44de-9039-d828bc27d430-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.422078 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.422053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.422186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.422107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.439254 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.439236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lckkb\" (UniqueName: \"kubernetes.io/projected/43182a8d-c31e-4b95-94e5-9e94ca06d6af-kube-api-access-lckkb\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.442642 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.442622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg7bf\" (UniqueName: \"kubernetes.io/projected/b3ac56f6-e1a0-4358-a163-9ea0f22a91a8-kube-api-access-kg7bf\") pod \"kube-state-metrics-84c885658c-qpqth\" (UID: \"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.442851 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.442833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf7jx\" (UniqueName: \"kubernetes.io/projected/08297acf-4eb9-44de-9039-d828bc27d430-kube-api-access-mf7jx\") pod \"openshift-state-metrics-86c5bc79d6-xj5qz\" (UID: \"08297acf-4eb9-44de-9039-d828bc27d430\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.477466 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.477446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"
Sep 17 17:35:26.489597 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.489578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth"
Sep 17 17:35:26.524888 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.524856    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:26.617399 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.617367    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:26.793320 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.793266    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz"]
Sep 17 17:35:26.795386 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:26.795361    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08297acf_4eb9_44de_9039_d828bc27d430.slice/crio-4870bb0401b137575d5f3886256b07b227a489109cf9e8d1edfe4037ba5f577f WatchSource:0}: Error finding container 4870bb0401b137575d5f3886256b07b227a489109cf9e8d1edfe4037ba5f577f: Status 404 returned error can't find the container with id 4870bb0401b137575d5f3886256b07b227a489109cf9e8d1edfe4037ba5f577f
Sep 17 17:35:26.809337 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.809318    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-qpqth"]
Sep 17 17:35:26.811464 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:26.811438    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ac56f6_e1a0_4358_a163_9ea0f22a91a8.slice/crio-61b7466a691b07c356ae090664c7aa7ea3292e17e7c3881b8bed9f10eb6d1eea WatchSource:0}: Error finding container 61b7466a691b07c356ae090664c7aa7ea3292e17e7c3881b8bed9f10eb6d1eea: Status 404 returned error can't find the container with id 61b7466a691b07c356ae090664c7aa7ea3292e17e7c3881b8bed9f10eb6d1eea
Sep 17 17:35:26.925141 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.925118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-tls\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:26.927339 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:26.927322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43182a8d-c31e-4b95-94e5-9e94ca06d6af-node-exporter-tls\") pod \"node-exporter-xshpf\" (UID: \"43182a8d-c31e-4b95-94e5-9e94ca06d6af\") " pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:27.109917 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.109895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xshpf"
Sep 17 17:35:27.136741 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:27.136717    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43182a8d_c31e_4b95_94e5_9e94ca06d6af.slice/crio-a614bf0bf7839f6d7ae980e355e233b70f532f451e2033f4ba9cee05090d8ad6 WatchSource:0}: Error finding container a614bf0bf7839f6d7ae980e355e233b70f532f451e2033f4ba9cee05090d8ad6: Status 404 returned error can't find the container with id a614bf0bf7839f6d7ae980e355e233b70f532f451e2033f4ba9cee05090d8ad6
Sep 17 17:35:27.159603 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.159574    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:27.277657 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.277627    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:27.545207 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.545167    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:27.602614 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.602578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xshpf" event={"ID":"43182a8d-c31e-4b95-94e5-9e94ca06d6af","Type":"ContainerStarted","Data":"a614bf0bf7839f6d7ae980e355e233b70f532f451e2033f4ba9cee05090d8ad6"}
Sep 17 17:35:27.605208 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.605079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz" event={"ID":"08297acf-4eb9-44de-9039-d828bc27d430","Type":"ContainerStarted","Data":"61800205e725f6f7a503a9bc2a15c7747b38e8dc83fc2c3d40267c95946d8246"}
Sep 17 17:35:27.605208 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.605142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz" event={"ID":"08297acf-4eb9-44de-9039-d828bc27d430","Type":"ContainerStarted","Data":"047e35b43c026de6a4c1dd7138c36e8a3312b7b634260ebf74151d7927a35aa8"}
Sep 17 17:35:27.605208 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.605174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz" event={"ID":"08297acf-4eb9-44de-9039-d828bc27d430","Type":"ContainerStarted","Data":"4870bb0401b137575d5f3886256b07b227a489109cf9e8d1edfe4037ba5f577f"}
Sep 17 17:35:27.606916 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.606854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth" event={"ID":"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8","Type":"ContainerStarted","Data":"61b7466a691b07c356ae090664c7aa7ea3292e17e7c3881b8bed9f10eb6d1eea"}
Sep 17 17:35:27.641562 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.641533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:27.844139 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:27.844059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.102108 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.102019    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.165167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.165134    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"]
Sep 17 17:35:29.167812 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.167794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.170346 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.170329    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 17:35:29.170346 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.170340    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ntcxp\""
Sep 17 17:35:29.170514 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.170473    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 17:35:29.170628 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.170611    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 17:35:29.170628 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.170622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 17:35:29.171315 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.171303    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 17:35:29.171371 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.171303    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8j0u8daruthtu\""
Sep 17 17:35:29.179345 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.179325    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"]
Sep 17 17:35:29.182488 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.182469    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.244247 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.244225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.244334 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.244257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c23ba056-4962-463d-8c1e-590f4bfbefcb-metrics-client-ca\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.244334 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.244282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.244404 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.244347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.244443 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.244407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-grpc-tls\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.244443 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.244431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56pk\" (UniqueName: \"kubernetes.io/projected/c23ba056-4962-463d-8c1e-590f4bfbefcb-kube-api-access-c56pk\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.244504 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.244455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-tls\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.244504 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.244477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.342282 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.342259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.345210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.345190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.345271 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.345230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-grpc-tls\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.345271 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.345247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56pk\" (UniqueName: \"kubernetes.io/projected/c23ba056-4962-463d-8c1e-590f4bfbefcb-kube-api-access-c56pk\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.345334 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.345269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-tls\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.345334 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.345297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.345423 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.345349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.345423 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.345373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c23ba056-4962-463d-8c1e-590f4bfbefcb-metrics-client-ca\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.345423 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.345406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.346236 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.346210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c23ba056-4962-463d-8c1e-590f4bfbefcb-metrics-client-ca\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.347830 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.347809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-tls\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.348139 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.348119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-grpc-tls\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.348206 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.348180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.348206 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.348197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.348275 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.348263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.348309 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.348287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c23ba056-4962-463d-8c1e-590f4bfbefcb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.355383 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.355340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56pk\" (UniqueName: \"kubernetes.io/projected/c23ba056-4962-463d-8c1e-590f4bfbefcb-kube-api-access-c56pk\") pod \"thanos-querier-5f55c86c89-2xk5g\" (UID: \"c23ba056-4962-463d-8c1e-590f4bfbefcb\") " pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.476613 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.476591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:29.560353 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.560330    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.619146 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.619079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth" event={"ID":"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8","Type":"ContainerStarted","Data":"f18b131956c74ea5963a77de919c0e8c266d79e4886d3b4c905c485c0eb54904"}
Sep 17 17:35:29.619263 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.619173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth" event={"ID":"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8","Type":"ContainerStarted","Data":"af262406965e5b2c5f9c3b3e9657fdb09f264c360ecd731596c7ac52bb54812f"}
Sep 17 17:35:29.619263 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.619207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth" event={"ID":"b3ac56f6-e1a0-4358-a163-9ea0f22a91a8","Type":"ContainerStarted","Data":"31de7d96a78c5dcb24258a7cf7ecc6b45cdb5d1d14da8d10d51c46bae21f2a74"}
Sep 17 17:35:29.620629 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.620606    2581 generic.go:356] "Generic (PLEG): container finished" podID="43182a8d-c31e-4b95-94e5-9e94ca06d6af" containerID="ba22b9fe1c9294cdb7fa6bc5f1a7606d5b5c77bdaf902881336438467a362929" exitCode=0
Sep 17 17:35:29.620721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.620667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xshpf" event={"ID":"43182a8d-c31e-4b95-94e5-9e94ca06d6af","Type":"ContainerDied","Data":"ba22b9fe1c9294cdb7fa6bc5f1a7606d5b5c77bdaf902881336438467a362929"}
Sep 17 17:35:29.622783 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.622763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz" event={"ID":"08297acf-4eb9-44de-9039-d828bc27d430","Type":"ContainerStarted","Data":"651704b308234e47987792a72d6f5877cd3143d67103478f6b04e193514549c2"}
Sep 17 17:35:29.641316 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.641278    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-qpqth" podStartSLOduration=1.90553992 podStartE2EDuration="3.641264343s" totalImagesPullingTime="1.735724423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:29.639332947 +0000 UTC m=+92.563804899" watchObservedRunningTime="2026-09-17 17:35:29.641264343 +0000 UTC m=+92.565736302"
Sep 17 17:35:29.663381 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.663344    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-xj5qz" podStartSLOduration=2.161752088 podStartE2EDuration="3.663330705s" totalImagesPullingTime="1.501578617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:29.662901027 +0000 UTC m=+92.587372986" watchObservedRunningTime="2026-09-17 17:35:29.663330705 +0000 UTC m=+92.587802664"
Sep 17 17:35:29.752074 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:29.752053    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"]
Sep 17 17:35:29.753839 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:29.753817    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23ba056_4962_463d_8c1e_590f4bfbefcb.slice/crio-0703458b1fd21288df11ebeaa05c7ecbec77fe95b7277ba39ab4a1c32228b243 WatchSource:0}: Error finding container 0703458b1fd21288df11ebeaa05c7ecbec77fe95b7277ba39ab4a1c32228b243: Status 404 returned error can't find the container with id 0703458b1fd21288df11ebeaa05c7ecbec77fe95b7277ba39ab4a1c32228b243
Sep 17 17:35:30.628876 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.628840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xshpf" event={"ID":"43182a8d-c31e-4b95-94e5-9e94ca06d6af","Type":"ContainerStarted","Data":"4a1fef4e1f4353f71fb6d4f308d18bd254bd4e53b740a81624c3d23b2bf134d7"}
Sep 17 17:35:30.629266 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.628901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xshpf" event={"ID":"43182a8d-c31e-4b95-94e5-9e94ca06d6af","Type":"ContainerStarted","Data":"d8eefb8ca4e96e631137cb1b13e492f44f8e6a6ce18d148917f1463958cbb411"}
Sep 17 17:35:30.630190 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.630162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g" event={"ID":"c23ba056-4962-463d-8c1e-590f4bfbefcb","Type":"ContainerStarted","Data":"0703458b1fd21288df11ebeaa05c7ecbec77fe95b7277ba39ab4a1c32228b243"}
Sep 17 17:35:30.643601 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.643578    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85cc868f54-p4pmd"]
Sep 17 17:35:30.646693 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.646677    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.649500 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.649474    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rtv44\""
Sep 17 17:35:30.649619 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.649547    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 17:35:30.650642 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.650621    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 17:35:30.650642 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.650639    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 17:35:30.650792 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.650645    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-61etu6asvfs5f\""
Sep 17 17:35:30.650792 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.650624    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:35:30.653862 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.653574    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xshpf" podStartSLOduration=3.151270504 podStartE2EDuration="4.653561103s" totalImagesPullingTime="1.411418327s" totalInitContainerRuntime="90.872272ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:30.651960513 +0000 UTC m=+93.576432473" watchObservedRunningTime="2026-09-17 17:35:30.653561103 +0000 UTC m=+93.578033112"
Sep 17 17:35:30.657574 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.657556    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85cc868f54-p4pmd"]
Sep 17 17:35:30.756834 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.756803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjpf6\" (UniqueName: \"kubernetes.io/projected/4091c712-43c0-43a7-a915-c3f4ff2c8276-kube-api-access-rjpf6\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.757212 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.757186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4091c712-43c0-43a7-a915-c3f4ff2c8276-secret-metrics-server-client-certs\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.757320 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.757236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4091c712-43c0-43a7-a915-c3f4ff2c8276-audit-log\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.757408 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.757387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4091c712-43c0-43a7-a915-c3f4ff2c8276-secret-metrics-server-tls\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.757467 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.757447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4091c712-43c0-43a7-a915-c3f4ff2c8276-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.757524 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.757497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4091c712-43c0-43a7-a915-c3f4ff2c8276-client-ca-bundle\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.757589 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.757551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4091c712-43c0-43a7-a915-c3f4ff2c8276-metrics-server-audit-profiles\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.858680 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.858566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4091c712-43c0-43a7-a915-c3f4ff2c8276-secret-metrics-server-tls\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.858680 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.858598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4091c712-43c0-43a7-a915-c3f4ff2c8276-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.858680 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.858626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4091c712-43c0-43a7-a915-c3f4ff2c8276-client-ca-bundle\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.858990 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.858800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4091c712-43c0-43a7-a915-c3f4ff2c8276-metrics-server-audit-profiles\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.858990 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.858865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjpf6\" (UniqueName: \"kubernetes.io/projected/4091c712-43c0-43a7-a915-c3f4ff2c8276-kube-api-access-rjpf6\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.858990 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.858906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4091c712-43c0-43a7-a915-c3f4ff2c8276-secret-metrics-server-client-certs\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.858990 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.858945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4091c712-43c0-43a7-a915-c3f4ff2c8276-audit-log\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.859293 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.859271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4091c712-43c0-43a7-a915-c3f4ff2c8276-audit-log\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.859396 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.859371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4091c712-43c0-43a7-a915-c3f4ff2c8276-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.860316 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.860181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4091c712-43c0-43a7-a915-c3f4ff2c8276-metrics-server-audit-profiles\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.861820 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.861802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4091c712-43c0-43a7-a915-c3f4ff2c8276-secret-metrics-server-tls\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.862900 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.862880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4091c712-43c0-43a7-a915-c3f4ff2c8276-secret-metrics-server-client-certs\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.863209 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.863189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4091c712-43c0-43a7-a915-c3f4ff2c8276-client-ca-bundle\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.869004 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.868983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjpf6\" (UniqueName: \"kubernetes.io/projected/4091c712-43c0-43a7-a915-c3f4ff2c8276-kube-api-access-rjpf6\") pod \"metrics-server-85cc868f54-p4pmd\" (UID: \"4091c712-43c0-43a7-a915-c3f4ff2c8276\") " pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:30.903337 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.903274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:30.957103 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:30.957076    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:31.209306 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:31.209232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:31.238180 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:31.238156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85cc868f54-p4pmd"]
Sep 17 17:35:31.240498 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:31.240452    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4091c712_43c0_43a7_a915_c3f4ff2c8276.slice/crio-4562c75c1e53682f979878b79704abca46a1ceb52069aaafff44312094ceabb1 WatchSource:0}: Error finding container 4562c75c1e53682f979878b79704abca46a1ceb52069aaafff44312094ceabb1: Status 404 returned error can't find the container with id 4562c75c1e53682f979878b79704abca46a1ceb52069aaafff44312094ceabb1
Sep 17 17:35:31.635758 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:31.635679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd" event={"ID":"4091c712-43c0-43a7-a915-c3f4ff2c8276","Type":"ContainerStarted","Data":"4562c75c1e53682f979878b79704abca46a1ceb52069aaafff44312094ceabb1"}
Sep 17 17:35:32.415415 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:32.415365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:32.649195 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:32.649133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g" event={"ID":"c23ba056-4962-463d-8c1e-590f4bfbefcb","Type":"ContainerStarted","Data":"e5b80a19180d74975a1433c483a834d70b9d0f723bd53b7f5e47eb939183c4bb"}
Sep 17 17:35:32.649588 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:32.649241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g" event={"ID":"c23ba056-4962-463d-8c1e-590f4bfbefcb","Type":"ContainerStarted","Data":"6e4f2cd3dd94122f2d9ac9aa5f8bed080d17c7e842af43bf53ba89389b671687"}
Sep 17 17:35:32.649588 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:32.649275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g" event={"ID":"c23ba056-4962-463d-8c1e-590f4bfbefcb","Type":"ContainerStarted","Data":"e88e2da0953b60678a5a3ee5f44ea144a3cbd890768cc32d8376d76f7ceefc8c"}
Sep 17 17:35:33.655643 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:33.655615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd" event={"ID":"4091c712-43c0-43a7-a915-c3f4ff2c8276","Type":"ContainerStarted","Data":"0f3451f5cc97e139861bd8ff33436ab63960a82ed58fbee14d6e8238ac57d1a5"}
Sep 17 17:35:33.658509 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:33.658483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g" event={"ID":"c23ba056-4962-463d-8c1e-590f4bfbefcb","Type":"ContainerStarted","Data":"b33e0fda4b56ad24f184f4c7783e8e04cf78b0eceea8510a93bdc58c05f7f2d0"}
Sep 17 17:35:33.658623 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:33.658539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g" event={"ID":"c23ba056-4962-463d-8c1e-590f4bfbefcb","Type":"ContainerStarted","Data":"d50b9b979d09ea4c2d431d02202c3ff3833c71909647ff8668aaaabd106c625b"}
Sep 17 17:35:33.658623 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:33.658565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g" event={"ID":"c23ba056-4962-463d-8c1e-590f4bfbefcb","Type":"ContainerStarted","Data":"1d6bcc077fb5ef1f31771de5e9ec8bc07d3dba2ec78315fc747623f3ecd113fe"}
Sep 17 17:35:33.658623 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:33.658594    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:33.692021 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:33.691981    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd" podStartSLOduration=1.765507474 podStartE2EDuration="3.691953064s" totalImagesPullingTime="1.92644559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:33.690621131 +0000 UTC m=+96.615093090" watchObservedRunningTime="2026-09-17 17:35:33.691953064 +0000 UTC m=+96.616425022"
Sep 17 17:35:33.727942 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:33.727899    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g" podStartSLOduration=1.495914887 podStartE2EDuration="4.727885339s" totalImagesPullingTime="3.231970452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:33.725762256 +0000 UTC m=+96.650234225" watchObservedRunningTime="2026-09-17 17:35:33.727885339 +0000 UTC m=+96.652357297"
Sep 17 17:35:35.555056 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:35.555021    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:35.655266 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:35.655242    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:36.504790 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:36.504758    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z485x"
Sep 17 17:35:36.922938 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:36.922859    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-8g78l"]
Sep 17 17:35:36.925815 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:36.925800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-8g78l"
Sep 17 17:35:36.928680 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:36.928654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dlg2k\""
Sep 17 17:35:36.928813 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:36.928691    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:35:36.928813 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:36.928656    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:35:36.936178 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:36.936160    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-8g78l"]
Sep 17 17:35:36.954803 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:36.954781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:37.006661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.006638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc88n\" (UniqueName: \"kubernetes.io/projected/a5fdaada-7867-4a8f-b7ee-9971ef25fbf7-kube-api-access-hc88n\") pod \"downloads-6594f4c75-8g78l\" (UID: \"a5fdaada-7867-4a8f-b7ee-9971ef25fbf7\") " pod="openshift-console/downloads-6594f4c75-8g78l"
Sep 17 17:35:37.049501 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.049479    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:37.107208 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.107189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc88n\" (UniqueName: \"kubernetes.io/projected/a5fdaada-7867-4a8f-b7ee-9971ef25fbf7-kube-api-access-hc88n\") pod \"downloads-6594f4c75-8g78l\" (UID: \"a5fdaada-7867-4a8f-b7ee-9971ef25fbf7\") " pod="openshift-console/downloads-6594f4c75-8g78l"
Sep 17 17:35:37.120371 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.120346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc88n\" (UniqueName: \"kubernetes.io/projected/a5fdaada-7867-4a8f-b7ee-9971ef25fbf7-kube-api-access-hc88n\") pod \"downloads-6594f4c75-8g78l\" (UID: \"a5fdaada-7867-4a8f-b7ee-9971ef25fbf7\") " pod="openshift-console/downloads-6594f4c75-8g78l"
Sep 17 17:35:37.216157 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.216124    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:37.235019 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.234994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-8g78l"
Sep 17 17:35:37.523151 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.523076    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-8g78l"]
Sep 17 17:35:37.526359 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.526335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:37.528239 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:37.528215    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fdaada_7867_4a8f_b7ee_9971ef25fbf7.slice/crio-0f44330ce51308f5b1afe43cdf9466ea2750582c652d8172de61bfc2b4b6d73b WatchSource:0}: Error finding container 0f44330ce51308f5b1afe43cdf9466ea2750582c652d8172de61bfc2b4b6d73b: Status 404 returned error can't find the container with id 0f44330ce51308f5b1afe43cdf9466ea2750582c652d8172de61bfc2b4b6d73b
Sep 17 17:35:37.664476 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.664448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:37.675785 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.675756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-8g78l" event={"ID":"a5fdaada-7867-4a8f-b7ee-9971ef25fbf7","Type":"ContainerStarted","Data":"0f44330ce51308f5b1afe43cdf9466ea2750582c652d8172de61bfc2b4b6d73b"}
Sep 17 17:35:37.740334 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.740308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:37.977662 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:37.977624    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:38.796924 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:38.796895    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:38.933533 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:38.933504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:39.032190 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:39.032155    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:39.622195 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:39.622160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:39.670780 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:39.670754    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f55c86c89-2xk5g"
Sep 17 17:35:39.683442 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:39.683395    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:42.004183 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:42.004150    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:42.123175 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:42.123146    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:42.220942 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:42.220909    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:43.182296 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:43.182263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:43.293399 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:43.293361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:45.627513 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:45.627479    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:45.695430 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:45.695403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:46.583268 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.583233    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:46.664451 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.664397    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:46.841641 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.841556    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78bc6459f9-prxn4"]
Sep 17 17:35:46.846054 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.846015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:46.848997 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.848958    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n8jkl\""
Sep 17 17:35:46.849136 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.849024    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:35:46.849136 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.849086    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:35:46.850169 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.850151    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:35:46.850297 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.850152    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:35:46.850577 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.850558    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:35:46.856793 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.856774    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bc6459f9-prxn4"]
Sep 17 17:35:46.989371 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.989338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mss2\" (UniqueName: \"kubernetes.io/projected/cead526c-1950-4fe2-91fa-22108776b58a-kube-api-access-4mss2\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:46.989526 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.989394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-console-config\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:46.989526 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.989423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-service-ca\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:46.989526 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.989466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-serving-cert\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:46.989670 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.989520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-oauth-serving-cert\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:46.989670 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:46.989613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-oauth-config\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.090508 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.090478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mss2\" (UniqueName: \"kubernetes.io/projected/cead526c-1950-4fe2-91fa-22108776b58a-kube-api-access-4mss2\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.090680 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.090527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-console-config\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.090680 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.090545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-service-ca\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.090680 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.090571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-serving-cert\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.090680 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.090592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-oauth-serving-cert\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.090680 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.090629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-oauth-config\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.091327 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.091284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-console-config\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.091438 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.091325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-service-ca\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.091438 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.091348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-oauth-serving-cert\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.093238 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.093194    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-serving-cert\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.093561 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.093537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-oauth-config\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.102316 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.102292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mss2\" (UniqueName: \"kubernetes.io/projected/cead526c-1950-4fe2-91fa-22108776b58a-kube-api-access-4mss2\") pod \"console-78bc6459f9-prxn4\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") " pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.158493 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.158465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:35:47.266261 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.266232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:47.333204 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:47.333177    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:49.364649 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:49.364610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:49.434112 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:49.434077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:50.225475 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:50.225442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:50.285066 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:50.285028    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:50.957346 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:50.957319    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:50.957719 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:50.957377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:35:51.046312 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:51.046281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:51.151236 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:51.151204    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:53.319501 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:53.319478    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bc6459f9-prxn4"]
Sep 17 17:35:53.321790 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:53.321760    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcead526c_1950_4fe2_91fa_22108776b58a.slice/crio-e7b42db7ca40a010d0965f059f2b46237d9ba732f99090fc82b75be22a46880b WatchSource:0}: Error finding container e7b42db7ca40a010d0965f059f2b46237d9ba732f99090fc82b75be22a46880b: Status 404 returned error can't find the container with id e7b42db7ca40a010d0965f059f2b46237d9ba732f99090fc82b75be22a46880b
Sep 17 17:35:53.747161 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:53.747121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bc6459f9-prxn4" event={"ID":"cead526c-1950-4fe2-91fa-22108776b58a","Type":"ContainerStarted","Data":"e7b42db7ca40a010d0965f059f2b46237d9ba732f99090fc82b75be22a46880b"}
Sep 17 17:35:53.749152 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:53.749121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-8g78l" event={"ID":"a5fdaada-7867-4a8f-b7ee-9971ef25fbf7","Type":"ContainerStarted","Data":"f647c64e77abb0c81c026656373e1d49932d6eaa5799a395d9292985009b7218"}
Sep 17 17:35:53.749371 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:53.749353    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-8g78l"
Sep 17 17:35:53.760849 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:53.760824    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-8g78l"
Sep 17 17:35:53.770015 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:53.769955    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-8g78l" podStartSLOduration=2.171229125 podStartE2EDuration="17.769939049s" totalImagesPullingTime="15.598709924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:53.767864463 +0000 UTC m=+116.692336421" watchObservedRunningTime="2026-09-17 17:35:53.769939049 +0000 UTC m=+116.694411007"
Sep 17 17:35:54.430451 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:54.430405    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:54.505190 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:54.504898    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:55.021715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.021675    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-587874cfb4-h8pmc"]
Sep 17 17:35:55.048159 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.048125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-587874cfb4-h8pmc"]
Sep 17 17:35:55.048324 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.048224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.058961 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.058443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-oauth-config\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.058961 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.058606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-oauth-serving-cert\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.058961 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.058652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-serving-cert\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.058961 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.058677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-console-config\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.058961 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.058705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-trusted-ca-bundle\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.058961 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.058782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-service-ca\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.058961 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.058845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26fl\" (UniqueName: \"kubernetes.io/projected/7040c17f-c735-4a7b-804b-452f7734aa4a-kube-api-access-x26fl\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.061163 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.060787    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:35:55.159663 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.159621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-service-ca\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.159835 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.159714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26fl\" (UniqueName: \"kubernetes.io/projected/7040c17f-c735-4a7b-804b-452f7734aa4a-kube-api-access-x26fl\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.159835 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.159763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-oauth-config\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.159835 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.159801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-oauth-serving-cert\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.160022 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.159867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-serving-cert\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.160022 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.159894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-console-config\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.160022 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.159917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-trusted-ca-bundle\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.160554 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.160509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-service-ca\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.160827 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.160801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-trusted-ca-bundle\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.161134 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.161098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-oauth-serving-cert\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.161843 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.161681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-console-config\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.163732 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.163708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-serving-cert\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.164243 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.164200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-oauth-config\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.170867 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.170844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26fl\" (UniqueName: \"kubernetes.io/projected/7040c17f-c735-4a7b-804b-452f7734aa4a-kube-api-access-x26fl\") pod \"console-587874cfb4-h8pmc\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") " pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:55.369241 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:55.369170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:35:56.626658 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:56.626623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:56.705653 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:56.705561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:58.782782 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:58.782749    2581 generic.go:356] "Generic (PLEG): container finished" podID="4575b36c-ec44-42ed-b49f-3cd2920ed4d5" containerID="975885ddffb0da1ca3b1da670a1d07d230d8c280c4174b5594f61236a2b8a2b3" exitCode=0
Sep 17 17:35:58.783238 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:58.782831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-b47t9" event={"ID":"4575b36c-ec44-42ed-b49f-3cd2920ed4d5","Type":"ContainerDied","Data":"975885ddffb0da1ca3b1da670a1d07d230d8c280c4174b5594f61236a2b8a2b3"}
Sep 17 17:35:58.783398 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:58.783342    2581 scope.go:117] "RemoveContainer" containerID="975885ddffb0da1ca3b1da670a1d07d230d8c280c4174b5594f61236a2b8a2b3"
Sep 17 17:35:58.834953 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:58.834923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:58.935258 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:58.935188    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:58.950412 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:58.950382    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-587874cfb4-h8pmc"]
Sep 17 17:35:59.100238 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:35:59.100202    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7040c17f_c735_4a7b_804b_452f7734aa4a.slice/crio-3ad753fa373dde4b1d50d0564f0eefa234bfe3299329a3d5bb23fac0681c7648 WatchSource:0}: Error finding container 3ad753fa373dde4b1d50d0564f0eefa234bfe3299329a3d5bb23fac0681c7648: Status 404 returned error can't find the container with id 3ad753fa373dde4b1d50d0564f0eefa234bfe3299329a3d5bb23fac0681c7648
Sep 17 17:35:59.142194 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:59.142168    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:59.245534 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:59.245441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:59.341605 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:59.341569    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:59.790747 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:59.790713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-b47t9" event={"ID":"4575b36c-ec44-42ed-b49f-3cd2920ed4d5","Type":"ContainerStarted","Data":"047e57d733f2aa671f6f77f932fc3ba8eece4fb90156b7cd4a89481b91458a1c"}
Sep 17 17:35:59.793027 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:59.793000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bc6459f9-prxn4" event={"ID":"cead526c-1950-4fe2-91fa-22108776b58a","Type":"ContainerStarted","Data":"7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1"}
Sep 17 17:35:59.794999 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:59.794957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587874cfb4-h8pmc" event={"ID":"7040c17f-c735-4a7b-804b-452f7734aa4a","Type":"ContainerStarted","Data":"b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11"}
Sep 17 17:35:59.795102 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:59.795016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587874cfb4-h8pmc" event={"ID":"7040c17f-c735-4a7b-804b-452f7734aa4a","Type":"ContainerStarted","Data":"3ad753fa373dde4b1d50d0564f0eefa234bfe3299329a3d5bb23fac0681c7648"}
Sep 17 17:35:59.834410 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:59.834087    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-587874cfb4-h8pmc" podStartSLOduration=4.834069501 podStartE2EDuration="4.834069501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:59.83317782 +0000 UTC m=+122.757649841" watchObservedRunningTime="2026-09-17 17:35:59.834069501 +0000 UTC m=+122.758541459"
Sep 17 17:35:59.855870 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:35:59.855813    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78bc6459f9-prxn4" podStartSLOduration=8.039209016 podStartE2EDuration="13.855793502s" totalImagesPullingTime="5.816584486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:59.854354775 +0000 UTC m=+122.778826734" watchObservedRunningTime="2026-09-17 17:35:59.855793502 +0000 UTC m=+122.780265461"
Sep 17 17:36:00.639005 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:00.638955    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:04.367838 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:04.367796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:04.464976 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:04.464928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:05.370153 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:05.370120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:36:05.370578 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:05.370178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:36:05.374856 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:05.374832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:36:05.824492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:05.824468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:36:05.892038 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:05.892007    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bc6459f9-prxn4"]
Sep 17 17:36:06.016428 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:06.016401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:06.119795 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:06.119738    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:06.225011 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:06.224986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:07.159346 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:07.159313    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:36:08.972153 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:08.972118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:36:08.974497 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:08.974464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b795638-8c73-4961-9d78-d850444de74c-metrics-certs\") pod \"network-metrics-daemon-tdmfn\" (UID: \"3b795638-8c73-4961-9d78-d850444de74c\") " pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:36:08.997570 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:08.997546    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2hx6\""
Sep 17 17:36:09.004905 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:09.004885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tdmfn"
Sep 17 17:36:09.291353 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:09.291281    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tdmfn"]
Sep 17 17:36:09.293504 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:36:09.293480    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b795638_8c73_4961_9d78_d850444de74c.slice/crio-c365ea11cca3456a37839698981910970c2f18cfdf06fe3491cca1cedf8a1bc3 WatchSource:0}: Error finding container c365ea11cca3456a37839698981910970c2f18cfdf06fe3491cca1cedf8a1bc3: Status 404 returned error can't find the container with id c365ea11cca3456a37839698981910970c2f18cfdf06fe3491cca1cedf8a1bc3
Sep 17 17:36:09.839355 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:09.839327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tdmfn" event={"ID":"3b795638-8c73-4961-9d78-d850444de74c","Type":"ContainerStarted","Data":"c365ea11cca3456a37839698981910970c2f18cfdf06fe3491cca1cedf8a1bc3"}
Sep 17 17:36:10.844687 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:10.844623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tdmfn" event={"ID":"3b795638-8c73-4961-9d78-d850444de74c","Type":"ContainerStarted","Data":"d147ed4318de261971df30634500d5fa373a937364dba311334034be37c13f7f"}
Sep 17 17:36:10.845087 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:10.844720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tdmfn" event={"ID":"3b795638-8c73-4961-9d78-d850444de74c","Type":"ContainerStarted","Data":"9baffbde84db7a2c0bf3a42fda7e4d08e2d6457a2511f9c5e941a1a7f1e88ea3"}
Sep 17 17:36:10.865148 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:10.865101    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tdmfn" podStartSLOduration=130.602986686 podStartE2EDuration="2m11.865086103s" totalImagesPullingTime="1.262099417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:10.863380076 +0000 UTC m=+133.787852044" watchObservedRunningTime="2026-09-17 17:36:10.865086103 +0000 UTC m=+133.789558088"
Sep 17 17:36:10.962882 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:10.962861    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:36:10.966492 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:10.966467    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85cc868f54-p4pmd"
Sep 17 17:36:11.028534 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:11.028512    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:11.121708 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:11.121654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:12.855789 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:12.855756    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d2e9f8e-1384-4723-8f54-c35a8322d56f" containerID="cc7eab7aab1cce1cde0c539e68fe96325631220fb811bcd18b225ebeba8647bc" exitCode=0
Sep 17 17:36:12.856271 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:12.855805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89" event={"ID":"9d2e9f8e-1384-4723-8f54-c35a8322d56f","Type":"ContainerDied","Data":"cc7eab7aab1cce1cde0c539e68fe96325631220fb811bcd18b225ebeba8647bc"}
Sep 17 17:36:12.856271 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:12.856154    2581 scope.go:117] "RemoveContainer" containerID="cc7eab7aab1cce1cde0c539e68fe96325631220fb811bcd18b225ebeba8647bc"
Sep 17 17:36:13.862451 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:13.862410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-97f89" event={"ID":"9d2e9f8e-1384-4723-8f54-c35a8322d56f","Type":"ContainerStarted","Data":"4e5eeb3eedc5ed2de7111afb5e3e71b0b73806290ee2dbfb98b2f73897f37004"}
Sep 17 17:36:14.868159 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:14.868125    2581 generic.go:356] "Generic (PLEG): container finished" podID="c894143e-e9a8-49a9-9e93-0de501d60d3b" containerID="ca7926fe543b39928d84730ccbb76b14d4be4783966ef47ec6b19883fd7d7336" exitCode=0
Sep 17 17:36:14.868528 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:14.868200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d" event={"ID":"c894143e-e9a8-49a9-9e93-0de501d60d3b","Type":"ContainerDied","Data":"ca7926fe543b39928d84730ccbb76b14d4be4783966ef47ec6b19883fd7d7336"}
Sep 17 17:36:14.868575 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:14.868564    2581 scope.go:117] "RemoveContainer" containerID="ca7926fe543b39928d84730ccbb76b14d4be4783966ef47ec6b19883fd7d7336"
Sep 17 17:36:15.873733 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:15.873699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4k49d" event={"ID":"c894143e-e9a8-49a9-9e93-0de501d60d3b","Type":"ContainerStarted","Data":"86e207c7114740ec1eef9b7ad3bc131da49f733ec378a6923579b7f5fc77efc6"}
Sep 17 17:36:19.346868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:19.346832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:19.404298 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:19.404266    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:26.181770 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:26.181736    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:26.240880 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:26.240858    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:30.910720 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:30.910680    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78bc6459f9-prxn4" podUID="cead526c-1950-4fe2-91fa-22108776b58a" containerName="console" containerID="cri-o://7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1" gracePeriod=15
Sep 17 17:36:31.690256 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.690232    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:36:31.827996 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.827907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-console-config\") pod \"cead526c-1950-4fe2-91fa-22108776b58a\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") "
Sep 17 17:36:31.827996 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.827986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-oauth-config\") pod \"cead526c-1950-4fe2-91fa-22108776b58a\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") "
Sep 17 17:36:31.828186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.828021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-oauth-serving-cert\") pod \"cead526c-1950-4fe2-91fa-22108776b58a\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") "
Sep 17 17:36:31.828186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.828078    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mss2\" (UniqueName: \"kubernetes.io/projected/cead526c-1950-4fe2-91fa-22108776b58a-kube-api-access-4mss2\") pod \"cead526c-1950-4fe2-91fa-22108776b58a\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") "
Sep 17 17:36:31.828186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.828134    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-serving-cert\") pod \"cead526c-1950-4fe2-91fa-22108776b58a\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") "
Sep 17 17:36:31.828343 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.828245    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-service-ca\") pod \"cead526c-1950-4fe2-91fa-22108776b58a\" (UID: \"cead526c-1950-4fe2-91fa-22108776b58a\") "
Sep 17 17:36:31.828343 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.828257    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-console-config" (OuterVolumeSpecName: "console-config") pod "cead526c-1950-4fe2-91fa-22108776b58a" (UID: "cead526c-1950-4fe2-91fa-22108776b58a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:31.828448 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.828401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cead526c-1950-4fe2-91fa-22108776b58a" (UID: "cead526c-1950-4fe2-91fa-22108776b58a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:31.828530 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.828513    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-console-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:36:31.828574 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.828534    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-oauth-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:36:31.828610 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.828593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-service-ca" (OuterVolumeSpecName: "service-ca") pod "cead526c-1950-4fe2-91fa-22108776b58a" (UID: "cead526c-1950-4fe2-91fa-22108776b58a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:31.830222 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.830199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cead526c-1950-4fe2-91fa-22108776b58a" (UID: "cead526c-1950-4fe2-91fa-22108776b58a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:31.830604 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.830575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cead526c-1950-4fe2-91fa-22108776b58a-kube-api-access-4mss2" (OuterVolumeSpecName: "kube-api-access-4mss2") pod "cead526c-1950-4fe2-91fa-22108776b58a" (UID: "cead526c-1950-4fe2-91fa-22108776b58a"). InnerVolumeSpecName "kube-api-access-4mss2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:31.830686 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.830639    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cead526c-1950-4fe2-91fa-22108776b58a" (UID: "cead526c-1950-4fe2-91fa-22108776b58a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:31.929324 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.929298    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-oauth-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:36:31.929324 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.929323    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mss2\" (UniqueName: \"kubernetes.io/projected/cead526c-1950-4fe2-91fa-22108776b58a-kube-api-access-4mss2\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:36:31.929629 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.929338    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cead526c-1950-4fe2-91fa-22108776b58a-console-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:36:31.929629 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.929351    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cead526c-1950-4fe2-91fa-22108776b58a-service-ca\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:36:31.938411 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.938384    2581 generic.go:356] "Generic (PLEG): container finished" podID="cead526c-1950-4fe2-91fa-22108776b58a" containerID="7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1" exitCode=0
Sep 17 17:36:31.938514 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.938436    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bc6459f9-prxn4"
Sep 17 17:36:31.938514 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.938466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bc6459f9-prxn4" event={"ID":"cead526c-1950-4fe2-91fa-22108776b58a","Type":"ContainerDied","Data":"7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1"}
Sep 17 17:36:31.938600 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.938522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bc6459f9-prxn4" event={"ID":"cead526c-1950-4fe2-91fa-22108776b58a","Type":"ContainerDied","Data":"e7b42db7ca40a010d0965f059f2b46237d9ba732f99090fc82b75be22a46880b"}
Sep 17 17:36:31.938600 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.938545    2581 scope.go:117] "RemoveContainer" containerID="7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1"
Sep 17 17:36:31.947340 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.947322    2581 scope.go:117] "RemoveContainer" containerID="7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1"
Sep 17 17:36:31.947613 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:36:31.947594    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1\": container with ID starting with 7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1 not found: ID does not exist" containerID="7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1"
Sep 17 17:36:31.947678 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.947621    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1"} err="failed to get container status \"7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1\": rpc error: code = NotFound desc = could not find container \"7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1\": container with ID starting with 7728899d4c8ff0d6c7d31a886da85d67dd2738e32468b6b221b4a04a04513fb1 not found: ID does not exist"
Sep 17 17:36:31.963678 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.963658    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bc6459f9-prxn4"]
Sep 17 17:36:31.972257 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:31.972238    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78bc6459f9-prxn4"]
Sep 17 17:36:33.179844 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:33.179811    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cead526c-1950-4fe2-91fa-22108776b58a" path="/var/lib/kubelet/pods/cead526c-1950-4fe2-91fa-22108776b58a/volumes"
Sep 17 17:36:34.354369 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:34.354330    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:34.452110 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:34.452078    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:45.448512 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:45.448417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:45.884587 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:45.884482    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:54.373925 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.372248    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75d4d7477-4klc4"]
Sep 17 17:36:54.373925 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.372677    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cead526c-1950-4fe2-91fa-22108776b58a" containerName="console"
Sep 17 17:36:54.373925 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.372692    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cead526c-1950-4fe2-91fa-22108776b58a" containerName="console"
Sep 17 17:36:54.373925 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.372793    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cead526c-1950-4fe2-91fa-22108776b58a" containerName="console"
Sep 17 17:36:54.375350 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.375332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.388916 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.388892    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d4d7477-4klc4"]
Sep 17 17:36:54.445546 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.445510    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:54.510110 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.510087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:54.517128 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.517105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-oauth-config\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.517235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.517144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-service-ca\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.517235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.517171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-serving-cert\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.517235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.517195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-trusted-ca-bundle\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.517235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.517217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvqr\" (UniqueName: \"kubernetes.io/projected/3cb82715-8649-455b-aaf6-d37d7eea618f-kube-api-access-jmvqr\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.517367 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.517256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-console-config\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.517367 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.517288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-oauth-serving-cert\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.617599 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.617574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-oauth-config\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.617730 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.617610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-service-ca\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.617730 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.617631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-serving-cert\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.617730 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.617692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-trusted-ca-bundle\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.617857 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.617743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvqr\" (UniqueName: \"kubernetes.io/projected/3cb82715-8649-455b-aaf6-d37d7eea618f-kube-api-access-jmvqr\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.617857 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.617815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-console-config\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.617933 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.617893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-oauth-serving-cert\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.618703 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.618670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-console-config\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.618822 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.618700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-oauth-serving-cert\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.618822 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.618745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-service-ca\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.618822 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.618773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-trusted-ca-bundle\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.620589 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.620294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-oauth-config\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.620589 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.620552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-serving-cert\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.627105 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.627049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvqr\" (UniqueName: \"kubernetes.io/projected/3cb82715-8649-455b-aaf6-d37d7eea618f-kube-api-access-jmvqr\") pod \"console-75d4d7477-4klc4\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") " pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.684077 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.684054    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:36:54.963445 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:54.963372    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d4d7477-4klc4"]
Sep 17 17:36:54.967137 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:36:54.967114    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb82715_8649_455b_aaf6_d37d7eea618f.slice/crio-96ab3e3572901d83ae88520104f73368ef7ef0886d9dc8519d793f8cf59d1b48 WatchSource:0}: Error finding container 96ab3e3572901d83ae88520104f73368ef7ef0886d9dc8519d793f8cf59d1b48: Status 404 returned error can't find the container with id 96ab3e3572901d83ae88520104f73368ef7ef0886d9dc8519d793f8cf59d1b48
Sep 17 17:36:55.037873 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:55.037842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d4d7477-4klc4" event={"ID":"3cb82715-8649-455b-aaf6-d37d7eea618f","Type":"ContainerStarted","Data":"893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf"}
Sep 17 17:36:55.038051 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:55.037898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d4d7477-4klc4" event={"ID":"3cb82715-8649-455b-aaf6-d37d7eea618f","Type":"ContainerStarted","Data":"96ab3e3572901d83ae88520104f73368ef7ef0886d9dc8519d793f8cf59d1b48"}
Sep 17 17:36:55.057468 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:36:55.057413    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75d4d7477-4klc4" podStartSLOduration=1.057391104 podStartE2EDuration="1.057391104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:55.055622346 +0000 UTC m=+177.980094306" watchObservedRunningTime="2026-09-17 17:36:55.057391104 +0000 UTC m=+177.981863064"
Sep 17 17:37:04.684530 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:04.684496    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:37:04.685010 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:04.684550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:37:04.689404 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:04.689385    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:37:05.083789 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:05.083765    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:37:05.144469 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:05.144436    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-587874cfb4-h8pmc"]
Sep 17 17:37:05.257333 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:05.257303    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:05.320145 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:05.320123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:05.382366 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:05.382297    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:07.051191 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:07.051153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:30.163741 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.163673    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-587874cfb4-h8pmc" podUID="7040c17f-c735-4a7b-804b-452f7734aa4a" containerName="console" containerID="cri-o://b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11" gracePeriod=15
Sep 17 17:37:30.402019 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.401996    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:37:30.486167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486137    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-oauth-config\") pod \"7040c17f-c735-4a7b-804b-452f7734aa4a\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") "
Sep 17 17:37:30.486167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486169    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-console-config\") pod \"7040c17f-c735-4a7b-804b-452f7734aa4a\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") "
Sep 17 17:37:30.486379 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486191    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x26fl\" (UniqueName: \"kubernetes.io/projected/7040c17f-c735-4a7b-804b-452f7734aa4a-kube-api-access-x26fl\") pod \"7040c17f-c735-4a7b-804b-452f7734aa4a\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") "
Sep 17 17:37:30.486379 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486235    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-serving-cert\") pod \"7040c17f-c735-4a7b-804b-452f7734aa4a\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") "
Sep 17 17:37:30.486379 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-service-ca\") pod \"7040c17f-c735-4a7b-804b-452f7734aa4a\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") "
Sep 17 17:37:30.486379 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486282    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-oauth-serving-cert\") pod \"7040c17f-c735-4a7b-804b-452f7734aa4a\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") "
Sep 17 17:37:30.486379 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486338    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-trusted-ca-bundle\") pod \"7040c17f-c735-4a7b-804b-452f7734aa4a\" (UID: \"7040c17f-c735-4a7b-804b-452f7734aa4a\") "
Sep 17 17:37:30.486742 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486717    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-service-ca" (OuterVolumeSpecName: "service-ca") pod "7040c17f-c735-4a7b-804b-452f7734aa4a" (UID: "7040c17f-c735-4a7b-804b-452f7734aa4a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:30.486805 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7040c17f-c735-4a7b-804b-452f7734aa4a" (UID: "7040c17f-c735-4a7b-804b-452f7734aa4a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:30.486941 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486908    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7040c17f-c735-4a7b-804b-452f7734aa4a" (UID: "7040c17f-c735-4a7b-804b-452f7734aa4a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:30.487082 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.486978    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-console-config" (OuterVolumeSpecName: "console-config") pod "7040c17f-c735-4a7b-804b-452f7734aa4a" (UID: "7040c17f-c735-4a7b-804b-452f7734aa4a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:30.488453 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.488430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7040c17f-c735-4a7b-804b-452f7734aa4a" (UID: "7040c17f-c735-4a7b-804b-452f7734aa4a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:30.488541 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.488472    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7040c17f-c735-4a7b-804b-452f7734aa4a" (UID: "7040c17f-c735-4a7b-804b-452f7734aa4a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:30.488581 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.488542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7040c17f-c735-4a7b-804b-452f7734aa4a-kube-api-access-x26fl" (OuterVolumeSpecName: "kube-api-access-x26fl") pod "7040c17f-c735-4a7b-804b-452f7734aa4a" (UID: "7040c17f-c735-4a7b-804b-452f7734aa4a"). InnerVolumeSpecName "kube-api-access-x26fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:30.587366 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.587330    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-trusted-ca-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:37:30.587366 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.587357    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-oauth-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:37:30.587366 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.587366    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-console-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:37:30.587366 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.587374    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x26fl\" (UniqueName: \"kubernetes.io/projected/7040c17f-c735-4a7b-804b-452f7734aa4a-kube-api-access-x26fl\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:37:30.587624 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.587385    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7040c17f-c735-4a7b-804b-452f7734aa4a-console-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:37:30.587624 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.587394    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-service-ca\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:37:30.587624 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:30.587402    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7040c17f-c735-4a7b-804b-452f7734aa4a-oauth-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:37:31.184190 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:31.184150    2581 generic.go:356] "Generic (PLEG): container finished" podID="7040c17f-c735-4a7b-804b-452f7734aa4a" containerID="b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11" exitCode=0
Sep 17 17:37:31.184568 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:31.184210    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587874cfb4-h8pmc"
Sep 17 17:37:31.184568 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:31.184229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587874cfb4-h8pmc" event={"ID":"7040c17f-c735-4a7b-804b-452f7734aa4a","Type":"ContainerDied","Data":"b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11"}
Sep 17 17:37:31.184568 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:31.184283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587874cfb4-h8pmc" event={"ID":"7040c17f-c735-4a7b-804b-452f7734aa4a","Type":"ContainerDied","Data":"3ad753fa373dde4b1d50d0564f0eefa234bfe3299329a3d5bb23fac0681c7648"}
Sep 17 17:37:31.184568 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:31.184304    2581 scope.go:117] "RemoveContainer" containerID="b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11"
Sep 17 17:37:31.192994 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:31.192953    2581 scope.go:117] "RemoveContainer" containerID="b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11"
Sep 17 17:37:31.193257 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:37:31.193242    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11\": container with ID starting with b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11 not found: ID does not exist" containerID="b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11"
Sep 17 17:37:31.193324 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:31.193263    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11"} err="failed to get container status \"b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11\": rpc error: code = NotFound desc = could not find container \"b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11\": container with ID starting with b38873351b3d00f106435c6caf943df1c1ce80c096fe4a26dd6c75a47f8ffb11 not found: ID does not exist"
Sep 17 17:37:31.208801 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:31.208775    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-587874cfb4-h8pmc"]
Sep 17 17:37:31.217259 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:31.217239    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-587874cfb4-h8pmc"]
Sep 17 17:37:33.180234 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:33.180201    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7040c17f-c735-4a7b-804b-452f7734aa4a" path="/var/lib/kubelet/pods/7040c17f-c735-4a7b-804b-452f7734aa4a/volumes"
Sep 17 17:37:43.942366 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:37:43.942333    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:00.658127 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:00.658096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:00.759802 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:00.759772    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:12.663502 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:12.663425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:12.730432 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:12.730406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:19.353600 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:19.353564    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:19.458720 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:19.458695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:20.740624 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.740588    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b9f899d44-qdcr9"]
Sep 17 17:38:20.741018 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.741003    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7040c17f-c735-4a7b-804b-452f7734aa4a" containerName="console"
Sep 17 17:38:20.741067 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.741021    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7040c17f-c735-4a7b-804b-452f7734aa4a" containerName="console"
Sep 17 17:38:20.741109 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.741100    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7040c17f-c735-4a7b-804b-452f7734aa4a" containerName="console"
Sep 17 17:38:20.743639 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.743621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.755445 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.755421    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b9f899d44-qdcr9"]
Sep 17 17:38:20.803698 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.803674    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:20.861432 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.861394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-trusted-ca-bundle\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.861594 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.861441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-oauth-serving-cert\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.861594 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.861468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-oauth-config\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.861594 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.861540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-service-ca\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.861729 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.861595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-serving-cert\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.861729 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.861621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndnk7\" (UniqueName: \"kubernetes.io/projected/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-kube-api-access-ndnk7\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.861729 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.861680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-config\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.944061 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.944020    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:20.962936 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.962912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-trusted-ca-bundle\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.963064 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.962944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-oauth-serving-cert\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.963064 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.962960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-oauth-config\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.963064 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.963009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-service-ca\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.963064 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.963045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-serving-cert\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.963252 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.963068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndnk7\" (UniqueName: \"kubernetes.io/projected/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-kube-api-access-ndnk7\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.963252 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.963237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-config\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.963695 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.963666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-oauth-serving-cert\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.963873 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.963849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-service-ca\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.964025 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.963886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-trusted-ca-bundle\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.964025 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.963927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-config\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.965349 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.965329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-oauth-config\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.965522 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.965504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-serving-cert\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:20.971843 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:20.971823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndnk7\" (UniqueName: \"kubernetes.io/projected/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-kube-api-access-ndnk7\") pod \"console-b9f899d44-qdcr9\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") " pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:21.011496 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:21.011407    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:21.052715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:21.052684    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:21.327060 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:21.326993    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b9f899d44-qdcr9"]
Sep 17 17:38:21.329960 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:38:21.329930    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7aeb81d_4002_45dd_8d7a_538ddc2836aa.slice/crio-b5cbed273bd14e6cfd221732a1f2bbfaa9b501f98929ca2476080e33dbf358c6 WatchSource:0}: Error finding container b5cbed273bd14e6cfd221732a1f2bbfaa9b501f98929ca2476080e33dbf358c6: Status 404 returned error can't find the container with id b5cbed273bd14e6cfd221732a1f2bbfaa9b501f98929ca2476080e33dbf358c6
Sep 17 17:38:21.380622 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:21.380598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9f899d44-qdcr9" event={"ID":"b7aeb81d-4002-45dd-8d7a-538ddc2836aa","Type":"ContainerStarted","Data":"b5cbed273bd14e6cfd221732a1f2bbfaa9b501f98929ca2476080e33dbf358c6"}
Sep 17 17:38:21.547131 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:21.547100    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:21.648631 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:21.648568    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:22.386776 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:22.386738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9f899d44-qdcr9" event={"ID":"b7aeb81d-4002-45dd-8d7a-538ddc2836aa","Type":"ContainerStarted","Data":"1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694"}
Sep 17 17:38:22.409006 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:22.408949    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b9f899d44-qdcr9" podStartSLOduration=2.408935356 podStartE2EDuration="2.408935356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:22.407257734 +0000 UTC m=+265.331729693" watchObservedRunningTime="2026-09-17 17:38:22.408935356 +0000 UTC m=+265.333407315"
Sep 17 17:38:31.053566 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:31.053529    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:31.054042 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:31.053587    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:31.058550 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:31.058521    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:31.431900 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:31.431815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:38:31.485947 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:31.485915    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d4d7477-4klc4"]
Sep 17 17:38:31.601206 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:31.601183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:31.702610 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:31.702584    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:31.762528 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:31.762499    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:56.507926 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.507883    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75d4d7477-4klc4" podUID="3cb82715-8649-455b-aaf6-d37d7eea618f" containerName="console" containerID="cri-o://893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf" gracePeriod=15
Sep 17 17:38:56.737380 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.737359    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:38:56.805088 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805024    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-trusted-ca-bundle\") pod \"3cb82715-8649-455b-aaf6-d37d7eea618f\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") "
Sep 17 17:38:56.805088 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-console-config\") pod \"3cb82715-8649-455b-aaf6-d37d7eea618f\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") "
Sep 17 17:38:56.805253 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-oauth-config\") pod \"3cb82715-8649-455b-aaf6-d37d7eea618f\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") "
Sep 17 17:38:56.805253 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmvqr\" (UniqueName: \"kubernetes.io/projected/3cb82715-8649-455b-aaf6-d37d7eea618f-kube-api-access-jmvqr\") pod \"3cb82715-8649-455b-aaf6-d37d7eea618f\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") "
Sep 17 17:38:56.805253 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805192    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-oauth-serving-cert\") pod \"3cb82715-8649-455b-aaf6-d37d7eea618f\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") "
Sep 17 17:38:56.805253 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805219    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-serving-cert\") pod \"3cb82715-8649-455b-aaf6-d37d7eea618f\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") "
Sep 17 17:38:56.805253 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-service-ca\") pod \"3cb82715-8649-455b-aaf6-d37d7eea618f\" (UID: \"3cb82715-8649-455b-aaf6-d37d7eea618f\") "
Sep 17 17:38:56.805542 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805512    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-console-config" (OuterVolumeSpecName: "console-config") pod "3cb82715-8649-455b-aaf6-d37d7eea618f" (UID: "3cb82715-8649-455b-aaf6-d37d7eea618f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:56.805542 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805528    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3cb82715-8649-455b-aaf6-d37d7eea618f" (UID: "3cb82715-8649-455b-aaf6-d37d7eea618f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:56.805659 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3cb82715-8649-455b-aaf6-d37d7eea618f" (UID: "3cb82715-8649-455b-aaf6-d37d7eea618f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:56.805780 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.805761    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-service-ca" (OuterVolumeSpecName: "service-ca") pod "3cb82715-8649-455b-aaf6-d37d7eea618f" (UID: "3cb82715-8649-455b-aaf6-d37d7eea618f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:56.807222 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.807199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3cb82715-8649-455b-aaf6-d37d7eea618f" (UID: "3cb82715-8649-455b-aaf6-d37d7eea618f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:56.807627 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.807611    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3cb82715-8649-455b-aaf6-d37d7eea618f" (UID: "3cb82715-8649-455b-aaf6-d37d7eea618f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:56.807699 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.807638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb82715-8649-455b-aaf6-d37d7eea618f-kube-api-access-jmvqr" (OuterVolumeSpecName: "kube-api-access-jmvqr") pod "3cb82715-8649-455b-aaf6-d37d7eea618f" (UID: "3cb82715-8649-455b-aaf6-d37d7eea618f"). InnerVolumeSpecName "kube-api-access-jmvqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:56.906514 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.906487    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-oauth-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:38:56.906514 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.906510    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:38:56.906638 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.906519    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-service-ca\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:38:56.906638 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.906528    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-trusted-ca-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:38:56.906638 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.906537    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cb82715-8649-455b-aaf6-d37d7eea618f-console-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:38:56.906638 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.906545    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cb82715-8649-455b-aaf6-d37d7eea618f-console-oauth-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:38:56.906638 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:56.906553    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvqr\" (UniqueName: \"kubernetes.io/projected/3cb82715-8649-455b-aaf6-d37d7eea618f-kube-api-access-jmvqr\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:38:57.542487 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:57.542457    2581 generic.go:356] "Generic (PLEG): container finished" podID="3cb82715-8649-455b-aaf6-d37d7eea618f" containerID="893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf" exitCode=0
Sep 17 17:38:57.542864 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:57.542500    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d4d7477-4klc4"
Sep 17 17:38:57.542864 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:57.542536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d4d7477-4klc4" event={"ID":"3cb82715-8649-455b-aaf6-d37d7eea618f","Type":"ContainerDied","Data":"893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf"}
Sep 17 17:38:57.542864 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:57.542573    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d4d7477-4klc4" event={"ID":"3cb82715-8649-455b-aaf6-d37d7eea618f","Type":"ContainerDied","Data":"96ab3e3572901d83ae88520104f73368ef7ef0886d9dc8519d793f8cf59d1b48"}
Sep 17 17:38:57.542864 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:57.542602    2581 scope.go:117] "RemoveContainer" containerID="893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf"
Sep 17 17:38:57.551267 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:57.551247    2581 scope.go:117] "RemoveContainer" containerID="893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf"
Sep 17 17:38:57.551540 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:38:57.551524    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf\": container with ID starting with 893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf not found: ID does not exist" containerID="893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf"
Sep 17 17:38:57.551588 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:57.551548    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf"} err="failed to get container status \"893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf\": rpc error: code = NotFound desc = could not find container \"893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf\": container with ID starting with 893358a57bf946840d1ca7b31c7ecfd4f6cffad68c6d399a36a19f87c7349adf not found: ID does not exist"
Sep 17 17:38:57.560322 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:57.560296    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d4d7477-4klc4"]
Sep 17 17:38:57.564885 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:57.564857    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75d4d7477-4klc4"]
Sep 17 17:38:59.079838 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:59.079807    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:38:59.082661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:59.082639    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:38:59.087115 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:59.087098    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:38:59.180671 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:38:59.180542    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb82715-8649-455b-aaf6-d37d7eea618f" path="/var/lib/kubelet/pods/3cb82715-8649-455b-aaf6-d37d7eea618f/volumes"
Sep 17 17:40:32.455297 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:32.455252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:51.242372 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.242336    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b7c9d8857-hzgbz"]
Sep 17 17:40:51.242818 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.242639    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb82715-8649-455b-aaf6-d37d7eea618f" containerName="console"
Sep 17 17:40:51.242818 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.242651    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb82715-8649-455b-aaf6-d37d7eea618f" containerName="console"
Sep 17 17:40:51.242818 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.242720    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb82715-8649-455b-aaf6-d37d7eea618f" containerName="console"
Sep 17 17:40:51.244673 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.244653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.260480 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.260457    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7c9d8857-hzgbz"]
Sep 17 17:40:51.288757 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.288737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:51.298057 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.298040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a425cd1c-90e6-491d-a77e-a62a5fa87354-console-serving-cert\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.298121 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.298077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a425cd1c-90e6-491d-a77e-a62a5fa87354-console-oauth-config\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.298121 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.298095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-service-ca\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.298201 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.298137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-console-config\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.298201 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.298192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmffr\" (UniqueName: \"kubernetes.io/projected/a425cd1c-90e6-491d-a77e-a62a5fa87354-kube-api-access-nmffr\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.298269 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.298214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-oauth-serving-cert\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.298269 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.298238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-trusted-ca-bundle\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.346466 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.346444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:51.399286 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.399266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-oauth-serving-cert\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.399391 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.399296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-trusted-ca-bundle\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.399391 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.399324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a425cd1c-90e6-491d-a77e-a62a5fa87354-console-serving-cert\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.399391 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.399349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a425cd1c-90e6-491d-a77e-a62a5fa87354-console-oauth-config\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.399548 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.399455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-service-ca\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.399548 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.399492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-console-config\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.399548 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.399517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmffr\" (UniqueName: \"kubernetes.io/projected/a425cd1c-90e6-491d-a77e-a62a5fa87354-kube-api-access-nmffr\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.400094 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.400067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-oauth-serving-cert\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.400199 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.400131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-trusted-ca-bundle\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.400199 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.400172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-service-ca\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.400275 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.400204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a425cd1c-90e6-491d-a77e-a62a5fa87354-console-config\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.401715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.401694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a425cd1c-90e6-491d-a77e-a62a5fa87354-console-serving-cert\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.401852 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.401836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a425cd1c-90e6-491d-a77e-a62a5fa87354-console-oauth-config\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.409547 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.409528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmffr\" (UniqueName: \"kubernetes.io/projected/a425cd1c-90e6-491d-a77e-a62a5fa87354-kube-api-access-nmffr\") pod \"console-7b7c9d8857-hzgbz\" (UID: \"a425cd1c-90e6-491d-a77e-a62a5fa87354\") " pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.554989 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.554916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:40:51.830018 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:51.829990    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7c9d8857-hzgbz"]
Sep 17 17:40:51.832251 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:40:51.832227    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda425cd1c_90e6_491d_a77e_a62a5fa87354.slice/crio-fa69ad09f0ae8657aa625ccb28bd5b592a0badec6744624d8f2a9eaaf5eb157a WatchSource:0}: Error finding container fa69ad09f0ae8657aa625ccb28bd5b592a0badec6744624d8f2a9eaaf5eb157a: Status 404 returned error can't find the container with id fa69ad09f0ae8657aa625ccb28bd5b592a0badec6744624d8f2a9eaaf5eb157a
Sep 17 17:40:52.010145 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:52.010110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7c9d8857-hzgbz" event={"ID":"a425cd1c-90e6-491d-a77e-a62a5fa87354","Type":"ContainerStarted","Data":"b64e02f126768391d913a1a3ae53fd8259bed9f65295497ff91841ce12d03099"}
Sep 17 17:40:52.010279 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:52.010168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7c9d8857-hzgbz" event={"ID":"a425cd1c-90e6-491d-a77e-a62a5fa87354","Type":"ContainerStarted","Data":"fa69ad09f0ae8657aa625ccb28bd5b592a0badec6744624d8f2a9eaaf5eb157a"}
Sep 17 17:40:52.028905 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:52.028862    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b7c9d8857-hzgbz" podStartSLOduration=1.028847846 podStartE2EDuration="1.028847846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:52.027210538 +0000 UTC m=+414.951682522" watchObservedRunningTime="2026-09-17 17:40:52.028847846 +0000 UTC m=+414.953319803"
Sep 17 17:40:55.458119 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:40:55.458084    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:01.556019 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:01.555980    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:41:01.556495 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:01.556040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:41:01.561011 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:01.560982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:41:02.057813 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:02.057788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b7c9d8857-hzgbz"
Sep 17 17:41:02.111372 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:02.111340    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b9f899d44-qdcr9"]
Sep 17 17:41:02.211980 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:02.211939    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:02.315861 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:02.315794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:02.380022 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:02.379997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:20.442295 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:20.442201    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:27.130712 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.130639    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b9f899d44-qdcr9" podUID="b7aeb81d-4002-45dd-8d7a-538ddc2836aa" containerName="console" containerID="cri-o://1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694" gracePeriod=15
Sep 17 17:41:27.363200 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.363178    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:41:27.453524 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.453503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-trusted-ca-bundle\") pod \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") "
Sep 17 17:41:27.453643 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.453534    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-oauth-serving-cert\") pod \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") "
Sep 17 17:41:27.453643 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.453556    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-service-ca\") pod \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") "
Sep 17 17:41:27.453643 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.453586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-serving-cert\") pod \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") "
Sep 17 17:41:27.453816 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.453757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-config\") pod \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") "
Sep 17 17:41:27.453816 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.453804    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndnk7\" (UniqueName: \"kubernetes.io/projected/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-kube-api-access-ndnk7\") pod \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") "
Sep 17 17:41:27.453916 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.453856    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-oauth-config\") pod \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\" (UID: \"b7aeb81d-4002-45dd-8d7a-538ddc2836aa\") "
Sep 17 17:41:27.454020 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.453994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b7aeb81d-4002-45dd-8d7a-538ddc2836aa" (UID: "b7aeb81d-4002-45dd-8d7a-538ddc2836aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:27.454181 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.454099    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-config" (OuterVolumeSpecName: "console-config") pod "b7aeb81d-4002-45dd-8d7a-538ddc2836aa" (UID: "b7aeb81d-4002-45dd-8d7a-538ddc2836aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:27.454181 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.453944    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7aeb81d-4002-45dd-8d7a-538ddc2836aa" (UID: "b7aeb81d-4002-45dd-8d7a-538ddc2836aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:27.454181 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.454077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7aeb81d-4002-45dd-8d7a-538ddc2836aa" (UID: "b7aeb81d-4002-45dd-8d7a-538ddc2836aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:27.454181 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.454164    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-trusted-ca-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:27.454181 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.454176    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-oauth-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:27.454357 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.454190    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-service-ca\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:27.454357 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.454199    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:27.455824 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.455798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7aeb81d-4002-45dd-8d7a-538ddc2836aa" (UID: "b7aeb81d-4002-45dd-8d7a-538ddc2836aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:41:27.455902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.455852    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-kube-api-access-ndnk7" (OuterVolumeSpecName: "kube-api-access-ndnk7") pod "b7aeb81d-4002-45dd-8d7a-538ddc2836aa" (UID: "b7aeb81d-4002-45dd-8d7a-538ddc2836aa"). InnerVolumeSpecName "kube-api-access-ndnk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:41:27.455902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.455860    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7aeb81d-4002-45dd-8d7a-538ddc2836aa" (UID: "b7aeb81d-4002-45dd-8d7a-538ddc2836aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:41:27.554612 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.554578    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndnk7\" (UniqueName: \"kubernetes.io/projected/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-kube-api-access-ndnk7\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:27.554612 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.554613    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-oauth-config\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:27.554612 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:27.554622    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7aeb81d-4002-45dd-8d7a-538ddc2836aa-console-serving-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:28.164057 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:28.164022    2581 generic.go:356] "Generic (PLEG): container finished" podID="b7aeb81d-4002-45dd-8d7a-538ddc2836aa" containerID="1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694" exitCode=0
Sep 17 17:41:28.164495 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:28.164077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9f899d44-qdcr9" event={"ID":"b7aeb81d-4002-45dd-8d7a-538ddc2836aa","Type":"ContainerDied","Data":"1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694"}
Sep 17 17:41:28.164495 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:28.164081    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9f899d44-qdcr9"
Sep 17 17:41:28.164495 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:28.164123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9f899d44-qdcr9" event={"ID":"b7aeb81d-4002-45dd-8d7a-538ddc2836aa","Type":"ContainerDied","Data":"b5cbed273bd14e6cfd221732a1f2bbfaa9b501f98929ca2476080e33dbf358c6"}
Sep 17 17:41:28.164495 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:28.164148    2581 scope.go:117] "RemoveContainer" containerID="1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694"
Sep 17 17:41:28.173822 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:28.173804    2581 scope.go:117] "RemoveContainer" containerID="1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694"
Sep 17 17:41:28.174152 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:41:28.174131    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694\": container with ID starting with 1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694 not found: ID does not exist" containerID="1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694"
Sep 17 17:41:28.174195 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:28.174162    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694"} err="failed to get container status \"1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694\": rpc error: code = NotFound desc = could not find container \"1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694\": container with ID starting with 1ce15f9ffdec8ab8deff04c933b600dc6639b0dc91095b3e9a8734a6276c5694 not found: ID does not exist"
Sep 17 17:41:28.186079 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:28.186054    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b9f899d44-qdcr9"]
Sep 17 17:41:28.190374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:28.190350    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b9f899d44-qdcr9"]
Sep 17 17:41:29.180174 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:29.180145    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7aeb81d-4002-45dd-8d7a-538ddc2836aa" path="/var/lib/kubelet/pods/b7aeb81d-4002-45dd-8d7a-538ddc2836aa/volumes"
Sep 17 17:41:29.255074 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:29.255041    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:29.358784 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:29.358756    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:29.468636 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:29.468612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:30.251684 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.251654    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"]
Sep 17 17:41:30.252156 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.252138    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7aeb81d-4002-45dd-8d7a-538ddc2836aa" containerName="console"
Sep 17 17:41:30.252264 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.252159    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7aeb81d-4002-45dd-8d7a-538ddc2836aa" containerName="console"
Sep 17 17:41:30.252264 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.252253    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7aeb81d-4002-45dd-8d7a-538ddc2836aa" containerName="console"
Sep 17 17:41:30.255462 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.255443    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.257938 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.257919    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:41:30.259108 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.259094    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:41:30.259179 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.259105    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4lnmx\""
Sep 17 17:41:30.263745 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.263723    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"]
Sep 17 17:41:30.374826 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.374796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.374826 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.374830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.374951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.374867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgczw\" (UniqueName: \"kubernetes.io/projected/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-kube-api-access-tgczw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.475204 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.475180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgczw\" (UniqueName: \"kubernetes.io/projected/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-kube-api-access-tgczw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.475288 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.475232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.475288 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.475264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.475593 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.475577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.475637 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.475620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.485541 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.485514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgczw\" (UniqueName: \"kubernetes.io/projected/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-kube-api-access-tgczw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.565371 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.565320    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:30.832217 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.832152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"]
Sep 17 17:41:30.835789 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:41:30.835758    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e70a3a_418b_4ac5_b0f3_9acc279ad7ec.slice/crio-e18a7cdce031ff35a15353bdd142cb4a50b30abeb78462affe210243c6a8556c WatchSource:0}: Error finding container e18a7cdce031ff35a15353bdd142cb4a50b30abeb78462affe210243c6a8556c: Status 404 returned error can't find the container with id e18a7cdce031ff35a15353bdd142cb4a50b30abeb78462affe210243c6a8556c
Sep 17 17:41:30.838236 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:30.838219    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:31.179747 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:31.179690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg" event={"ID":"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec","Type":"ContainerStarted","Data":"e18a7cdce031ff35a15353bdd142cb4a50b30abeb78462affe210243c6a8556c"}
Sep 17 17:41:37.208982 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:37.208932    2581 generic.go:356] "Generic (PLEG): container finished" podID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerID="ff76c91d3f1bd2dc8f6aad2e28b0d3479663883f7e01e614045d22bb2c27ffda" exitCode=0
Sep 17 17:41:37.209312 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:37.209024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg" event={"ID":"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec","Type":"ContainerDied","Data":"ff76c91d3f1bd2dc8f6aad2e28b0d3479663883f7e01e614045d22bb2c27ffda"}
Sep 17 17:41:40.223379 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:40.223351    2581 generic.go:356] "Generic (PLEG): container finished" podID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerID="3c48090240655a05efab1ede59dc0c334e470447ffde75000a82f2ba64f94344" exitCode=0
Sep 17 17:41:40.223658 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:40.223389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg" event={"ID":"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec","Type":"ContainerDied","Data":"3c48090240655a05efab1ede59dc0c334e470447ffde75000a82f2ba64f94344"}
Sep 17 17:41:47.258036 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:47.258006    2581 generic.go:356] "Generic (PLEG): container finished" podID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerID="de6a9e64d07ae84da616877252b4de4c8efc7597bfbaaa6ce0e7b306baf0c8ec" exitCode=0
Sep 17 17:41:47.258410 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:47.258092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg" event={"ID":"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec","Type":"ContainerDied","Data":"de6a9e64d07ae84da616877252b4de4c8efc7597bfbaaa6ce0e7b306baf0c8ec"}
Sep 17 17:41:48.387905 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.387883    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:48.513140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.513119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgczw\" (UniqueName: \"kubernetes.io/projected/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-kube-api-access-tgczw\") pod \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") "
Sep 17 17:41:48.513249 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.513161    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-util\") pod \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") "
Sep 17 17:41:48.513249 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.513201    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-bundle\") pod \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\" (UID: \"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec\") "
Sep 17 17:41:48.513791 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.513769    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-bundle" (OuterVolumeSpecName: "bundle") pod "52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" (UID: "52e70a3a-418b-4ac5-b0f3-9acc279ad7ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:41:48.515253 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.515201    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-kube-api-access-tgczw" (OuterVolumeSpecName: "kube-api-access-tgczw") pod "52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" (UID: "52e70a3a-418b-4ac5-b0f3-9acc279ad7ec"). InnerVolumeSpecName "kube-api-access-tgczw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:41:48.517295 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.517273    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-util" (OuterVolumeSpecName: "util") pod "52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" (UID: "52e70a3a-418b-4ac5-b0f3-9acc279ad7ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:41:48.614215 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.614192    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:48.614215 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.614211    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:48.614329 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:48.614219    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgczw\" (UniqueName: \"kubernetes.io/projected/52e70a3a-418b-4ac5-b0f3-9acc279ad7ec-kube-api-access-tgczw\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:41:49.270383 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:49.270352    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg"
Sep 17 17:41:49.270515 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:49.270360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am6rtg" event={"ID":"52e70a3a-418b-4ac5-b0f3-9acc279ad7ec","Type":"ContainerDied","Data":"e18a7cdce031ff35a15353bdd142cb4a50b30abeb78462affe210243c6a8556c"}
Sep 17 17:41:49.270515 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:49.270474    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e18a7cdce031ff35a15353bdd142cb4a50b30abeb78462affe210243c6a8556c"
Sep 17 17:41:51.285662 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:51.285632    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:51.382148 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:51.382118    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.055501 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.055452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.325652 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.325575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.500073 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.500040    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.544484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.544459    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"]
Sep 17 17:41:57.544751 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.544740    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerName="util"
Sep 17 17:41:57.544792 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.544752    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerName="util"
Sep 17 17:41:57.544792 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.544767    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerName="pull"
Sep 17 17:41:57.544792 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.544773    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerName="pull"
Sep 17 17:41:57.544792 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.544786    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerName="extract"
Sep 17 17:41:57.544792 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.544793    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerName="extract"
Sep 17 17:41:57.544936 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.544852    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="52e70a3a-418b-4ac5-b0f3-9acc279ad7ec" containerName="extract"
Sep 17 17:41:57.548153 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.548136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:57.550899 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.550880    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 17:41:57.551007 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.550880    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 17:41:57.551007 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.550914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 17:41:57.552089 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.552073    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:41:57.552187 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.552103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-wrfqw\""
Sep 17 17:41:57.552187 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.552126    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:41:57.558697 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.558675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"]
Sep 17 17:41:57.581957 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.581909    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.642609 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.642586    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.675729 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.675709    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:57.675804 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.675751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:57.675804 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.675784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbt4x\" (UniqueName: \"kubernetes.io/projected/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-kube-api-access-sbt4x\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:57.781139 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.776616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbt4x\" (UniqueName: \"kubernetes.io/projected/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-kube-api-access-sbt4x\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:57.781139 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.776702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:57.781139 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.776752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:57.781139 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:41:57.776925    2581 secret.go:281] references non-existent secret key: tls.crt
Sep 17 17:41:57.781139 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:41:57.776942    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 17:41:57.781139 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:41:57.776961    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp: references non-existent secret key: tls.crt
Sep 17 17:41:57.781139 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:41:57.777043    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-certificates podName:d5f3d2e5-9c2b-442c-b83f-3e31bbf524db nodeName:}" failed. No retries permitted until 2026-09-17 17:41:58.277025439 +0000 UTC m=+481.201497376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-certificates") pod "keda-metrics-apiserver-77857b7b98-gclqp" (UID: "d5f3d2e5-9c2b-442c-b83f-3e31bbf524db") : references non-existent secret key: tls.crt
Sep 17 17:41:57.781139 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.777477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:57.786672 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:57.786644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbt4x\" (UniqueName: \"kubernetes.io/projected/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-kube-api-access-sbt4x\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:58.013263 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.013239    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.079623 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.079605    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-h5gxz"]
Sep 17 17:41:58.082091 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.082077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:41:58.084629 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.084611    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 17:41:58.091369 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.091345    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-h5gxz"]
Sep 17 17:41:58.113860 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.113839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.179715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.179692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2w5c\" (UniqueName: \"kubernetes.io/projected/98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156-kube-api-access-c2w5c\") pod \"keda-admission-6c4d864b86-h5gxz\" (UID: \"98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156\") " pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:41:58.179847 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.179733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156-certificates\") pod \"keda-admission-6c4d864b86-h5gxz\" (UID: \"98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156\") " pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:41:58.215537 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.215503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.280484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.280406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2w5c\" (UniqueName: \"kubernetes.io/projected/98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156-kube-api-access-c2w5c\") pod \"keda-admission-6c4d864b86-h5gxz\" (UID: \"98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156\") " pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:41:58.280484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.280476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156-certificates\") pod \"keda-admission-6c4d864b86-h5gxz\" (UID: \"98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156\") " pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:41:58.280647 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.280521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:58.284074 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.283488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5f3d2e5-9c2b-442c-b83f-3e31bbf524db-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gclqp\" (UID: \"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:58.285142 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.285121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156-certificates\") pod \"keda-admission-6c4d864b86-h5gxz\" (UID: \"98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156\") " pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:41:58.290785 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.290760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2w5c\" (UniqueName: \"kubernetes.io/projected/98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156-kube-api-access-c2w5c\") pod \"keda-admission-6c4d864b86-h5gxz\" (UID: \"98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156\") " pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:41:58.392879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.392855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:41:58.458606 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.458356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:41:58.459627 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.459562    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.657994 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.657888    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.665047 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.665023    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-h5gxz"]
Sep 17 17:41:58.667042 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:41:58.667018    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ca4a7d_9d53_4ee2_ab69_a1f6d52c9156.slice/crio-08b3a6f7f76ec86184a0964a9b0185569c948b47d77cd5d3b662f96934b9cc77 WatchSource:0}: Error finding container 08b3a6f7f76ec86184a0964a9b0185569c948b47d77cd5d3b662f96934b9cc77: Status 404 returned error can't find the container with id 08b3a6f7f76ec86184a0964a9b0185569c948b47d77cd5d3b662f96934b9cc77
Sep 17 17:41:58.734712 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.734691    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"]
Sep 17 17:41:58.736424 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:41:58.736399    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f3d2e5_9c2b_442c_b83f_3e31bbf524db.slice/crio-c6b949325e145b00e2f7e144c8af69de620960652d000cc3e1a45fe6b7d33b1a WatchSource:0}: Error finding container c6b949325e145b00e2f7e144c8af69de620960652d000cc3e1a45fe6b7d33b1a: Status 404 returned error can't find the container with id c6b949325e145b00e2f7e144c8af69de620960652d000cc3e1a45fe6b7d33b1a
Sep 17 17:41:58.781576 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.781553    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.886206 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:58.886183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.312974 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:59.312932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-h5gxz" event={"ID":"98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156","Type":"ContainerStarted","Data":"08b3a6f7f76ec86184a0964a9b0185569c948b47d77cd5d3b662f96934b9cc77"}
Sep 17 17:41:59.314270 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:41:59.314236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp" event={"ID":"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db","Type":"ContainerStarted","Data":"c6b949325e145b00e2f7e144c8af69de620960652d000cc3e1a45fe6b7d33b1a"}
Sep 17 17:42:01.325818 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:01.325784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp" event={"ID":"d5f3d2e5-9c2b-442c-b83f-3e31bbf524db","Type":"ContainerStarted","Data":"f9151f765f0e7bda263a8a014f02fd3233184f1b5e600d877abee18575f931b7"}
Sep 17 17:42:01.326254 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:01.325903    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:42:12.334702 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:12.334675    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp"
Sep 17 17:42:12.356981 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:12.356911    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gclqp" podStartSLOduration=13.279601172 podStartE2EDuration="15.35689761s" totalImagesPullingTime="2.077296438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:01.354641728 +0000 UTC m=+484.279113686" watchObservedRunningTime="2026-09-17 17:42:12.35689761 +0000 UTC m=+495.281369568"
Sep 17 17:42:12.381373 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:12.381344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-h5gxz" event={"ID":"98ca4a7d-9d53-4ee2-ab69-a1f6d52c9156","Type":"ContainerStarted","Data":"107de2bd69ac4914ef7981e3454214f5e4c2f7eed6d83bb924ee158be32aa37b"}
Sep 17 17:42:12.381527 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:12.381460    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:42:12.402052 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:12.401998    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-h5gxz" podStartSLOduration=1.740467061 podStartE2EDuration="14.401978131s" totalImagesPullingTime="12.66151107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:12.399274088 +0000 UTC m=+495.323746049" watchObservedRunningTime="2026-09-17 17:42:12.401978131 +0000 UTC m=+495.326450083"
Sep 17 17:42:33.388472 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:33.388434    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-h5gxz"
Sep 17 17:42:50.113365 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.113284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:50.217129 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.217097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:50.426541 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.426448    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"]
Sep 17 17:42:50.429322 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.429296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.432232 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.432208    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:42:50.432340 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.432258    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:42:50.433275 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.433250    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4lnmx\""
Sep 17 17:42:50.451237 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.451211    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"]
Sep 17 17:42:50.578739 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.578715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.578739 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.578741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.578885 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.578767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68rb\" (UniqueName: \"kubernetes.io/projected/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-kube-api-access-l68rb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.679079 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.679020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.679079 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.679045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.679079 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.679070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68rb\" (UniqueName: \"kubernetes.io/projected/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-kube-api-access-l68rb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.679478 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.679454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.679544 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.679491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.688723 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.688702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68rb\" (UniqueName: \"kubernetes.io/projected/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-kube-api-access-l68rb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:50.739163 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:50.739144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:51.016304 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:51.016282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"]
Sep 17 17:42:51.018865 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:42:51.018841    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fcec39d_2f9b_4563_96f0_e4d05acb6bea.slice/crio-5280bd32aa8441d3b46f3d721de81462e5592118d2ba6194365f8c40e87406b2 WatchSource:0}: Error finding container 5280bd32aa8441d3b46f3d721de81462e5592118d2ba6194365f8c40e87406b2: Status 404 returned error can't find the container with id 5280bd32aa8441d3b46f3d721de81462e5592118d2ba6194365f8c40e87406b2
Sep 17 17:42:51.560278 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:51.560243    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerID="5e56af0ef416e1fd95b8fb6de6c611a8fefeeab5d62e7062939cb456c671d880" exitCode=0
Sep 17 17:42:51.560610 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:51.560324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn" event={"ID":"0fcec39d-2f9b-4563-96f0-e4d05acb6bea","Type":"ContainerDied","Data":"5e56af0ef416e1fd95b8fb6de6c611a8fefeeab5d62e7062939cb456c671d880"}
Sep 17 17:42:51.560610 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:51.560385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn" event={"ID":"0fcec39d-2f9b-4563-96f0-e4d05acb6bea","Type":"ContainerStarted","Data":"5280bd32aa8441d3b46f3d721de81462e5592118d2ba6194365f8c40e87406b2"}
Sep 17 17:42:53.572370 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:53.572343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn" event={"ID":"0fcec39d-2f9b-4563-96f0-e4d05acb6bea","Type":"ContainerStarted","Data":"c1dfeddc0711a801a723c51e62deb79f56b6ead8c72897fad4c513b38c40a1d5"}
Sep 17 17:42:54.579276 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:54.579236    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerID="c1dfeddc0711a801a723c51e62deb79f56b6ead8c72897fad4c513b38c40a1d5" exitCode=0
Sep 17 17:42:54.579645 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:54.579331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn" event={"ID":"0fcec39d-2f9b-4563-96f0-e4d05acb6bea","Type":"ContainerDied","Data":"c1dfeddc0711a801a723c51e62deb79f56b6ead8c72897fad4c513b38c40a1d5"}
Sep 17 17:42:55.586476 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:55.586446    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerID="33b8589d3a781f7bf9cee829d6c574e46a8af4b19a5cde5a3519a1ed33a79d25" exitCode=0
Sep 17 17:42:55.586849 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:55.586533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn" event={"ID":"0fcec39d-2f9b-4563-96f0-e4d05acb6bea","Type":"ContainerDied","Data":"33b8589d3a781f7bf9cee829d6c574e46a8af4b19a5cde5a3519a1ed33a79d25"}
Sep 17 17:42:56.719859 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.719832    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:56.826993 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.826943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-bundle\") pod \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") "
Sep 17 17:42:56.827121 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.827039    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l68rb\" (UniqueName: \"kubernetes.io/projected/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-kube-api-access-l68rb\") pod \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") "
Sep 17 17:42:56.827121 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.827058    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-util\") pod \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\" (UID: \"0fcec39d-2f9b-4563-96f0-e4d05acb6bea\") "
Sep 17 17:42:56.827590 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.827565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-bundle" (OuterVolumeSpecName: "bundle") pod "0fcec39d-2f9b-4563-96f0-e4d05acb6bea" (UID: "0fcec39d-2f9b-4563-96f0-e4d05acb6bea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:56.829049 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.829027    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-kube-api-access-l68rb" (OuterVolumeSpecName: "kube-api-access-l68rb") pod "0fcec39d-2f9b-4563-96f0-e4d05acb6bea" (UID: "0fcec39d-2f9b-4563-96f0-e4d05acb6bea"). InnerVolumeSpecName "kube-api-access-l68rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:56.831329 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.831305    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-util" (OuterVolumeSpecName: "util") pod "0fcec39d-2f9b-4563-96f0-e4d05acb6bea" (UID: "0fcec39d-2f9b-4563-96f0-e4d05acb6bea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:56.928197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.928146    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:42:56.928197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.928169    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68rb\" (UniqueName: \"kubernetes.io/projected/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-kube-api-access-l68rb\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:42:56.928197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:56.928179    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fcec39d-2f9b-4563-96f0-e4d05acb6bea-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:42:57.598524 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:57.598494    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn"
Sep 17 17:42:57.598655 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:57.598490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bkqfn" event={"ID":"0fcec39d-2f9b-4563-96f0-e4d05acb6bea","Type":"ContainerDied","Data":"5280bd32aa8441d3b46f3d721de81462e5592118d2ba6194365f8c40e87406b2"}
Sep 17 17:42:57.598655 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:42:57.598615    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5280bd32aa8441d3b46f3d721de81462e5592118d2ba6194365f8c40e87406b2"
Sep 17 17:43:01.116712 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:01.116679    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:02.367418 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:02.367383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:04.735433 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:04.735391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:04.835057 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:04.835025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:07.213735 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:07.213698    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:07.328770 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:07.328741    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:09.429140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:09.429111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:09.529744 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:09.529718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:09.629518 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:09.629495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:10.049874 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.049847    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:10.985743 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.985704    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"]
Sep 17 17:43:10.986140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.986056    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerName="pull"
Sep 17 17:43:10.986140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.986067    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerName="pull"
Sep 17 17:43:10.986140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.986077    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerName="extract"
Sep 17 17:43:10.986140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.986082    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerName="extract"
Sep 17 17:43:10.986140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.986092    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerName="util"
Sep 17 17:43:10.986140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.986098    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerName="util"
Sep 17 17:43:10.986313 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.986164    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fcec39d-2f9b-4563-96f0-e4d05acb6bea" containerName="extract"
Sep 17 17:43:10.988387 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.988370    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"
Sep 17 17:43:10.992844 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.992823    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 17:43:10.994003 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.993982    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xxlpp\""
Sep 17 17:43:10.994098 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:10.994004    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 17:43:11.000047 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.000024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"]
Sep 17 17:43:11.130645 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.130622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14ff805c-e788-4c8e-be70-0aace5ca4227-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4n4dn\" (UID: \"14ff805c-e788-4c8e-be70-0aace5ca4227\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"
Sep 17 17:43:11.130730 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.130658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqwv\" (UniqueName: \"kubernetes.io/projected/14ff805c-e788-4c8e-be70-0aace5ca4227-kube-api-access-mnqwv\") pod \"cert-manager-cainjector-559cdd8c87-4n4dn\" (UID: \"14ff805c-e788-4c8e-be70-0aace5ca4227\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"
Sep 17 17:43:11.231519 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.231494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14ff805c-e788-4c8e-be70-0aace5ca4227-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4n4dn\" (UID: \"14ff805c-e788-4c8e-be70-0aace5ca4227\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"
Sep 17 17:43:11.231611 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.231531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqwv\" (UniqueName: \"kubernetes.io/projected/14ff805c-e788-4c8e-be70-0aace5ca4227-kube-api-access-mnqwv\") pod \"cert-manager-cainjector-559cdd8c87-4n4dn\" (UID: \"14ff805c-e788-4c8e-be70-0aace5ca4227\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"
Sep 17 17:43:11.241174 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.241121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqwv\" (UniqueName: \"kubernetes.io/projected/14ff805c-e788-4c8e-be70-0aace5ca4227-kube-api-access-mnqwv\") pod \"cert-manager-cainjector-559cdd8c87-4n4dn\" (UID: \"14ff805c-e788-4c8e-be70-0aace5ca4227\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"
Sep 17 17:43:11.241174 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.241138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14ff805c-e788-4c8e-be70-0aace5ca4227-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4n4dn\" (UID: \"14ff805c-e788-4c8e-be70-0aace5ca4227\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"
Sep 17 17:43:11.297372 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.297354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"
Sep 17 17:43:11.536251 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.536179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:11.566304 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.566280    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn"]
Sep 17 17:43:11.568510 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:43:11.568480    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14ff805c_e788_4c8e_be70_0aace5ca4227.slice/crio-1a5dec794930fd7147a99302354589b1e1797b36813f60ee76c134c9fcb9d0c4 WatchSource:0}: Error finding container 1a5dec794930fd7147a99302354589b1e1797b36813f60ee76c134c9fcb9d0c4: Status 404 returned error can't find the container with id 1a5dec794930fd7147a99302354589b1e1797b36813f60ee76c134c9fcb9d0c4
Sep 17 17:43:11.633688 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.633666    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:11.671017 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.670995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn" event={"ID":"14ff805c-e788-4c8e-be70-0aace5ca4227","Type":"ContainerStarted","Data":"1a5dec794930fd7147a99302354589b1e1797b36813f60ee76c134c9fcb9d0c4"}
Sep 17 17:43:11.735881 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.735857    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:11.841830 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.841779    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"]
Sep 17 17:43:11.844095 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.844078    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:11.846561 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.846543    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:43:11.847607 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.847589    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:43:11.847697 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.847591    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4lnmx\""
Sep 17 17:43:11.854922 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.854900    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"]
Sep 17 17:43:11.936188 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.936166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:11.936274 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.936214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:11.936310 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:11.936282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjf4f\" (UniqueName: \"kubernetes.io/projected/0ac339f4-b601-4a8c-bdf0-42525781b8c3-kube-api-access-wjf4f\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:12.037285 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.037260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:12.037599 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.037300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:12.037599 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.037354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjf4f\" (UniqueName: \"kubernetes.io/projected/0ac339f4-b601-4a8c-bdf0-42525781b8c3-kube-api-access-wjf4f\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:12.037716 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.037607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:12.037716 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.037700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:12.051529 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.051512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjf4f\" (UniqueName: \"kubernetes.io/projected/0ac339f4-b601-4a8c-bdf0-42525781b8c3-kube-api-access-wjf4f\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:12.153937 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.153891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:12.435829 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.435802    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"]
Sep 17 17:43:12.438191 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:43:12.438161    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac339f4_b601_4a8c_bdf0_42525781b8c3.slice/crio-ff3b8f1425da1e8876a34e9a02a72d40febcf4906a258059d91447da6355443f WatchSource:0}: Error finding container ff3b8f1425da1e8876a34e9a02a72d40febcf4906a258059d91447da6355443f: Status 404 returned error can't find the container with id ff3b8f1425da1e8876a34e9a02a72d40febcf4906a258059d91447da6355443f
Sep 17 17:43:12.678334 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.678295    2581 generic.go:356] "Generic (PLEG): container finished" podID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerID="37942934b0eb66e01ef402e4a61976b38de28958d36f0cf38b654f98344c00a5" exitCode=0
Sep 17 17:43:12.678479 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.678352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46" event={"ID":"0ac339f4-b601-4a8c-bdf0-42525781b8c3","Type":"ContainerDied","Data":"37942934b0eb66e01ef402e4a61976b38de28958d36f0cf38b654f98344c00a5"}
Sep 17 17:43:12.678479 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:12.678420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46" event={"ID":"0ac339f4-b601-4a8c-bdf0-42525781b8c3","Type":"ContainerStarted","Data":"ff3b8f1425da1e8876a34e9a02a72d40febcf4906a258059d91447da6355443f"}
Sep 17 17:43:14.690099 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:14.690064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn" event={"ID":"14ff805c-e788-4c8e-be70-0aace5ca4227","Type":"ContainerStarted","Data":"07efb13c843aad55322d03a4519914aeee9443f6fe2b7a2f866b45fd22c9c5ec"}
Sep 17 17:43:14.708611 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:14.708568    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4n4dn" podStartSLOduration=2.252688202 podStartE2EDuration="4.708554424s" totalImagesPullingTime="2.455866222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:14.706583268 +0000 UTC m=+557.631055230" watchObservedRunningTime="2026-09-17 17:43:14.708554424 +0000 UTC m=+557.633026382"
Sep 17 17:43:14.909003 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:14.908958    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:14.966458 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:14.966432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:15.697341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:15.697309    2581 generic.go:356] "Generic (PLEG): container finished" podID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerID="672e0b5b59a9f7b399ad63587eb0e0e76202f885582aef8788eec5d90ae4650c" exitCode=0
Sep 17 17:43:15.697701 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:15.697394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46" event={"ID":"0ac339f4-b601-4a8c-bdf0-42525781b8c3","Type":"ContainerDied","Data":"672e0b5b59a9f7b399ad63587eb0e0e76202f885582aef8788eec5d90ae4650c"}
Sep 17 17:43:16.705147 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:16.705107    2581 generic.go:356] "Generic (PLEG): container finished" podID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerID="52fa07ff2296dd6e31c8ac6b3797b9423566782939aed81c96374de6753b4497" exitCode=0
Sep 17 17:43:16.705551 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:16.705195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46" event={"ID":"0ac339f4-b601-4a8c-bdf0-42525781b8c3","Type":"ContainerDied","Data":"52fa07ff2296dd6e31c8ac6b3797b9423566782939aed81c96374de6753b4497"}
Sep 17 17:43:17.834226 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:17.834202    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:17.838824 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:17.838804    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:17.985617 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:17.985566    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjf4f\" (UniqueName: \"kubernetes.io/projected/0ac339f4-b601-4a8c-bdf0-42525781b8c3-kube-api-access-wjf4f\") pod \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") "
Sep 17 17:43:17.985617 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:17.985602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-bundle\") pod \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") "
Sep 17 17:43:17.985774 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:17.985634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-util\") pod \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\" (UID: \"0ac339f4-b601-4a8c-bdf0-42525781b8c3\") "
Sep 17 17:43:17.985956 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:17.985929    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-bundle" (OuterVolumeSpecName: "bundle") pod "0ac339f4-b601-4a8c-bdf0-42525781b8c3" (UID: "0ac339f4-b601-4a8c-bdf0-42525781b8c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:17.987583 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:17.987566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac339f4-b601-4a8c-bdf0-42525781b8c3-kube-api-access-wjf4f" (OuterVolumeSpecName: "kube-api-access-wjf4f") pod "0ac339f4-b601-4a8c-bdf0-42525781b8c3" (UID: "0ac339f4-b601-4a8c-bdf0-42525781b8c3"). InnerVolumeSpecName "kube-api-access-wjf4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:43:18.087133 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:18.087112    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjf4f\" (UniqueName: \"kubernetes.io/projected/0ac339f4-b601-4a8c-bdf0-42525781b8c3-kube-api-access-wjf4f\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:43:18.087133 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:18.087131    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:43:18.089270 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:18.089246    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-util" (OuterVolumeSpecName: "util") pod "0ac339f4-b601-4a8c-bdf0-42525781b8c3" (UID: "0ac339f4-b601-4a8c-bdf0-42525781b8c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:18.187843 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:18.187824    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ac339f4-b601-4a8c-bdf0-42525781b8c3-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:43:18.716845 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:18.716823    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46"
Sep 17 17:43:18.716845 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:18.716830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnhq46" event={"ID":"0ac339f4-b601-4a8c-bdf0-42525781b8c3","Type":"ContainerDied","Data":"ff3b8f1425da1e8876a34e9a02a72d40febcf4906a258059d91447da6355443f"}
Sep 17 17:43:18.716993 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:18.716865    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff3b8f1425da1e8876a34e9a02a72d40febcf4906a258059d91447da6355443f"
Sep 17 17:43:23.022929 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:23.022889    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:23.131108 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:23.131071    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:35.132702 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:35.132665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:40.023272 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:40.023236    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:40.107034 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:40.107010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:41.049210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:41.049171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:44.252017 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:44.251957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:44.353016 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:44.352982    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:46.572590 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.572548    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"]
Sep 17 17:43:46.573128 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.573107    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerName="util"
Sep 17 17:43:46.573197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.573133    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerName="util"
Sep 17 17:43:46.573197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.573148    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerName="pull"
Sep 17 17:43:46.573197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.573156    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerName="pull"
Sep 17 17:43:46.573197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.573177    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerName="extract"
Sep 17 17:43:46.573197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.573187    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerName="extract"
Sep 17 17:43:46.573408 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.573276    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ac339f4-b601-4a8c-bdf0-42525781b8c3" containerName="extract"
Sep 17 17:43:46.575720 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.575698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.578774 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.578679    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:43:46.578774 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.578700    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-gbzzr\""
Sep 17 17:43:46.578774 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.578716    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:43:46.578774 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.578737    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:43:46.587496 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.587473    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"]
Sep 17 17:43:46.593201 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.593179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f34990-ab55-4d22-b575-ae36e8990569-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.593302 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.593208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f34990-ab55-4d22-b575-ae36e8990569-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.593302 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.593235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f34990-ab55-4d22-b575-ae36e8990569-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.593405 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.593327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.593405 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.593361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.593405 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.593388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.593527 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.593411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.593527 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.593426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.593527 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.593443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsk8p\" (UniqueName: \"kubernetes.io/projected/67f34990-ab55-4d22-b575-ae36e8990569-kube-api-access-vsk8p\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.610087 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.610065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:46.686708 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.686681    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:46.694764 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.694741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.694826 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.694771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.694826 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.694786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.694826 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.694813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsk8p\" (UniqueName: \"kubernetes.io/projected/67f34990-ab55-4d22-b575-ae36e8990569-kube-api-access-vsk8p\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.694871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f34990-ab55-4d22-b575-ae36e8990569-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.694896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f34990-ab55-4d22-b575-ae36e8990569-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.694919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f34990-ab55-4d22-b575-ae36e8990569-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.694996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695203 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.695039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695203 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.695172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695311 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.695253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695365 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.695346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695440 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.695419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.695540 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.695519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f34990-ab55-4d22-b575-ae36e8990569-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.696957 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.696939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f34990-ab55-4d22-b575-ae36e8990569-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.697322 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.697304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f34990-ab55-4d22-b575-ae36e8990569-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.703487 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.703463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsk8p\" (UniqueName: \"kubernetes.io/projected/67f34990-ab55-4d22-b575-ae36e8990569-kube-api-access-vsk8p\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.703571 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.703483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f34990-ab55-4d22-b575-ae36e8990569-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg\" (UID: \"67f34990-ab55-4d22-b575-ae36e8990569\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:46.888212 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:46.888160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:47.165150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:47.165125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"]
Sep 17 17:43:47.167066 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:43:47.167038    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f34990_ab55_4d22_b575_ae36e8990569.slice/crio-f419973ccff324a932d3df0ca990ce2d63bed88680d5f262814b47fc5f1715f3 WatchSource:0}: Error finding container f419973ccff324a932d3df0ca990ce2d63bed88680d5f262814b47fc5f1715f3: Status 404 returned error can't find the container with id f419973ccff324a932d3df0ca990ce2d63bed88680d5f262814b47fc5f1715f3
Sep 17 17:43:47.860898 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:47.860855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" event={"ID":"67f34990-ab55-4d22-b575-ae36e8990569","Type":"ContainerStarted","Data":"f419973ccff324a932d3df0ca990ce2d63bed88680d5f262814b47fc5f1715f3"}
Sep 17 17:43:49.941453 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:49.941412    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:43:49.941754 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:49.941490    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:43:49.941754 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:49.941522    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:43:50.203689 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:50.203660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:50.879043 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:50.879007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" event={"ID":"67f34990-ab55-4d22-b575-ae36e8990569","Type":"ContainerStarted","Data":"9bdbcc8f96e18c6bd992cb0da0f67f284fe259d285bcf93235e091451ab577f8"}
Sep 17 17:43:50.889063 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:50.889035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:50.904822 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:50.904775    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" podStartSLOduration=2.1327787320000002 podStartE2EDuration="4.904760299s" totalImagesPullingTime="2.771981567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:50.902838411 +0000 UTC m=+593.827310369" watchObservedRunningTime="2026-09-17 17:43:50.904760299 +0000 UTC m=+593.829232257"
Sep 17 17:43:51.894074 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:51.894045    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:52.889311 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:52.889281    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:52.890500 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:52.890476    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:43:53.072296 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:53.072254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:53.175791 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:53.175726    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:56.156377 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.156337    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"]
Sep 17 17:43:56.160062 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.160041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.162901 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.162877    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:43:56.163896 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.163872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4lnmx\""
Sep 17 17:43:56.164025 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.163921    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:43:56.168501 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.168473    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"]
Sep 17 17:43:56.245103 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.245078    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"]
Sep 17 17:43:56.248291 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.248276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.257022 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.257001    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"]
Sep 17 17:43:56.265575 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.265548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8xv\" (UniqueName: \"kubernetes.io/projected/340c0480-49d8-433b-90b0-194dd10ef44f-kube-api-access-9x8xv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.265690 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.265595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.265760 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.265704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.345462 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.345439    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"]
Sep 17 17:43:56.348724 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.348705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.357397 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.357373    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"]
Sep 17 17:43:56.366715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.366177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8xv\" (UniqueName: \"kubernetes.io/projected/340c0480-49d8-433b-90b0-194dd10ef44f-kube-api-access-9x8xv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.366715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.366210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.366715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.366231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.366715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.366258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.366715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.366339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr8hc\" (UniqueName: \"kubernetes.io/projected/6ba3c671-4fdd-4fd9-93dc-57ce544748db-kube-api-access-cr8hc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.366715 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.366382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.366990 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.366763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.366990 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.366773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.374134 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.374109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8xv\" (UniqueName: \"kubernetes.io/projected/340c0480-49d8-433b-90b0-194dd10ef44f-kube-api-access-9x8xv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.450400 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.450371    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"]
Sep 17 17:43:56.453813 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.453800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.463879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.463862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"]
Sep 17 17:43:56.466865 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.466846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.466950 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.466877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.466950 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.466898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8hc\" (UniqueName: \"kubernetes.io/projected/6ba3c671-4fdd-4fd9-93dc-57ce544748db-kube-api-access-cr8hc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.466950 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.466923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.466950 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.466939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqtv\" (UniqueName: \"kubernetes.io/projected/ea71400a-4396-498b-9486-f543ac037955-kube-api-access-9xqtv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.467174 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.467079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.467274 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.467256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.467369 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.467356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.470765 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.470746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:43:56.475494 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.475471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8hc\" (UniqueName: \"kubernetes.io/projected/6ba3c671-4fdd-4fd9-93dc-57ce544748db-kube-api-access-cr8hc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.556835 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.556810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:43:56.568080 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.568048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwzf4\" (UniqueName: \"kubernetes.io/projected/68fb73d7-97b6-48da-bd21-32c181dc9a61-kube-api-access-dwzf4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.568129 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.568079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.568129 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.568103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.568206 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.568186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqtv\" (UniqueName: \"kubernetes.io/projected/ea71400a-4396-498b-9486-f543ac037955-kube-api-access-9xqtv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.568292 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.568271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.568404 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.568308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.568514 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.568481    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.568599 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.568525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.577203 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.577183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqtv\" (UniqueName: \"kubernetes.io/projected/ea71400a-4396-498b-9486-f543ac037955-kube-api-access-9xqtv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.659026 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.659003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:43:56.669165 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.669147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.669236 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.669173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.669236 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.669196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwzf4\" (UniqueName: \"kubernetes.io/projected/68fb73d7-97b6-48da-bd21-32c181dc9a61-kube-api-access-dwzf4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.669646 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.669618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.669646 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.669640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.679405 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.679385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwzf4\" (UniqueName: \"kubernetes.io/projected/68fb73d7-97b6-48da-bd21-32c181dc9a61-kube-api-access-dwzf4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.741709 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.741682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"]
Sep 17 17:43:56.743491 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:43:56.743467    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340c0480_49d8_433b_90b0_194dd10ef44f.slice/crio-22147903ba5fae12e549222f093bb3037078e3e40021b34e2ad26e6904ca2392 WatchSource:0}: Error finding container 22147903ba5fae12e549222f093bb3037078e3e40021b34e2ad26e6904ca2392: Status 404 returned error can't find the container with id 22147903ba5fae12e549222f093bb3037078e3e40021b34e2ad26e6904ca2392
Sep 17 17:43:56.762434 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.762413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:43:56.830397 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.830233    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"]
Sep 17 17:43:56.832320 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:43:56.832290    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba3c671_4fdd_4fd9_93dc_57ce544748db.slice/crio-bcc2e5644eae27484768ed2b7027684d9fc1242ac4f368f5f87f0ce47e1ae5a8 WatchSource:0}: Error finding container bcc2e5644eae27484768ed2b7027684d9fc1242ac4f368f5f87f0ce47e1ae5a8: Status 404 returned error can't find the container with id bcc2e5644eae27484768ed2b7027684d9fc1242ac4f368f5f87f0ce47e1ae5a8
Sep 17 17:43:56.910185 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.910154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p" event={"ID":"6ba3c671-4fdd-4fd9-93dc-57ce544748db","Type":"ContainerStarted","Data":"bcc2e5644eae27484768ed2b7027684d9fc1242ac4f368f5f87f0ce47e1ae5a8"}
Sep 17 17:43:56.911743 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.911717    2581 generic.go:356] "Generic (PLEG): container finished" podID="340c0480-49d8-433b-90b0-194dd10ef44f" containerID="18166b7b7ebd5b2b1bacc557c71a67eaca8d0300e2a637ceb890756a75969b08" exitCode=0
Sep 17 17:43:56.911855 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.911757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd" event={"ID":"340c0480-49d8-433b-90b0-194dd10ef44f","Type":"ContainerDied","Data":"18166b7b7ebd5b2b1bacc557c71a67eaca8d0300e2a637ceb890756a75969b08"}
Sep 17 17:43:56.911855 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.911793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd" event={"ID":"340c0480-49d8-433b-90b0-194dd10ef44f","Type":"ContainerStarted","Data":"22147903ba5fae12e549222f093bb3037078e3e40021b34e2ad26e6904ca2392"}
Sep 17 17:43:56.928693 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:56.928663    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"]
Sep 17 17:43:56.930631 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:43:56.930605    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea71400a_4396_498b_9486_f543ac037955.slice/crio-059b4b0fe1429e12923bfa3987e84e2ad7d6a0673c4d58f349e187c71fbe768a WatchSource:0}: Error finding container 059b4b0fe1429e12923bfa3987e84e2ad7d6a0673c4d58f349e187c71fbe768a: Status 404 returned error can't find the container with id 059b4b0fe1429e12923bfa3987e84e2ad7d6a0673c4d58f349e187c71fbe768a
Sep 17 17:43:57.034955 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.034932    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"]
Sep 17 17:43:57.082733 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:43:57.082708    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fb73d7_97b6_48da_bd21_32c181dc9a61.slice/crio-bc720abefcc160646f856369f56c0fb615f27349a2110bf468da90de08716237 WatchSource:0}: Error finding container bc720abefcc160646f856369f56c0fb615f27349a2110bf468da90de08716237: Status 404 returned error can't find the container with id bc720abefcc160646f856369f56c0fb615f27349a2110bf468da90de08716237
Sep 17 17:43:57.919148 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.919082    2581 generic.go:356] "Generic (PLEG): container finished" podID="ea71400a-4396-498b-9486-f543ac037955" containerID="87939749dd28f78cb20032af0b6e5f015d4116ba2466b1e36d95a6a121acbe14" exitCode=0
Sep 17 17:43:57.919484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.919172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9" event={"ID":"ea71400a-4396-498b-9486-f543ac037955","Type":"ContainerDied","Data":"87939749dd28f78cb20032af0b6e5f015d4116ba2466b1e36d95a6a121acbe14"}
Sep 17 17:43:57.919484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.919233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9" event={"ID":"ea71400a-4396-498b-9486-f543ac037955","Type":"ContainerStarted","Data":"059b4b0fe1429e12923bfa3987e84e2ad7d6a0673c4d58f349e187c71fbe768a"}
Sep 17 17:43:57.921238 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.921211    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerID="0e531e57a2350319e49ebf88d7eec03d887545306594e80de2e1fedf7e51c34b" exitCode=0
Sep 17 17:43:57.921358 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.921297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p" event={"ID":"6ba3c671-4fdd-4fd9-93dc-57ce544748db","Type":"ContainerDied","Data":"0e531e57a2350319e49ebf88d7eec03d887545306594e80de2e1fedf7e51c34b"}
Sep 17 17:43:57.923151 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.923101    2581 generic.go:356] "Generic (PLEG): container finished" podID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerID="40d25c4638ca876213d0ecd080a5723c191bb078f589a408561cb3bc022f456b" exitCode=0
Sep 17 17:43:57.923242 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.923198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl" event={"ID":"68fb73d7-97b6-48da-bd21-32c181dc9a61","Type":"ContainerDied","Data":"40d25c4638ca876213d0ecd080a5723c191bb078f589a408561cb3bc022f456b"}
Sep 17 17:43:57.923301 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.923266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl" event={"ID":"68fb73d7-97b6-48da-bd21-32c181dc9a61","Type":"ContainerStarted","Data":"bc720abefcc160646f856369f56c0fb615f27349a2110bf468da90de08716237"}
Sep 17 17:43:57.925663 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.925620    2581 generic.go:356] "Generic (PLEG): container finished" podID="340c0480-49d8-433b-90b0-194dd10ef44f" containerID="4d1c6747d0e0335f85971684914a24020309e4ffae0ae013015615d9d44db9d0" exitCode=0
Sep 17 17:43:57.925825 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:57.925671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd" event={"ID":"340c0480-49d8-433b-90b0-194dd10ef44f","Type":"ContainerDied","Data":"4d1c6747d0e0335f85971684914a24020309e4ffae0ae013015615d9d44db9d0"}
Sep 17 17:43:58.932806 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:58.932740    2581 generic.go:356] "Generic (PLEG): container finished" podID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerID="65c95f75f2d1f3e90df7934a56ee9bc40e5aeaef03eba581e9aed31f4d150e11" exitCode=0
Sep 17 17:43:58.933175 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:58.932824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl" event={"ID":"68fb73d7-97b6-48da-bd21-32c181dc9a61","Type":"ContainerDied","Data":"65c95f75f2d1f3e90df7934a56ee9bc40e5aeaef03eba581e9aed31f4d150e11"}
Sep 17 17:43:58.935242 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:58.935221    2581 generic.go:356] "Generic (PLEG): container finished" podID="340c0480-49d8-433b-90b0-194dd10ef44f" containerID="f9187ae30c871880921ce1d5c39bf9e44327dbdf9dae3a50035d88625405aa8f" exitCode=0
Sep 17 17:43:58.935347 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:58.935277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd" event={"ID":"340c0480-49d8-433b-90b0-194dd10ef44f","Type":"ContainerDied","Data":"f9187ae30c871880921ce1d5c39bf9e44327dbdf9dae3a50035d88625405aa8f"}
Sep 17 17:43:59.125436 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:59.125412    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:43:59.127107 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:59.127079    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:43:59.944112 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:59.944033    2581 generic.go:356] "Generic (PLEG): container finished" podID="ea71400a-4396-498b-9486-f543ac037955" containerID="5fff1d8afc470ec9a0b198597f0e3e28c2099b92c59dca19dc25a4f0866d60ef" exitCode=0
Sep 17 17:43:59.944565 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:59.944124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9" event={"ID":"ea71400a-4396-498b-9486-f543ac037955","Type":"ContainerDied","Data":"5fff1d8afc470ec9a0b198597f0e3e28c2099b92c59dca19dc25a4f0866d60ef"}
Sep 17 17:43:59.948027 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:59.947779    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerID="ae90312714eb6801988618f5dac888e569fa7408bcfc1163a6507103c7f4072f" exitCode=0
Sep 17 17:43:59.948027 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:59.947918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p" event={"ID":"6ba3c671-4fdd-4fd9-93dc-57ce544748db","Type":"ContainerDied","Data":"ae90312714eb6801988618f5dac888e569fa7408bcfc1163a6507103c7f4072f"}
Sep 17 17:43:59.952356 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:59.952329    2581 generic.go:356] "Generic (PLEG): container finished" podID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerID="2f5eed13a071129af8887a37d56aa9e8593c29956590244a400b8ea01a00a2e0" exitCode=0
Sep 17 17:43:59.952482 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:43:59.952395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl" event={"ID":"68fb73d7-97b6-48da-bd21-32c181dc9a61","Type":"ContainerDied","Data":"2f5eed13a071129af8887a37d56aa9e8593c29956590244a400b8ea01a00a2e0"}
Sep 17 17:44:00.093734 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.093710    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:44:00.208213 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.208194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x8xv\" (UniqueName: \"kubernetes.io/projected/340c0480-49d8-433b-90b0-194dd10ef44f-kube-api-access-9x8xv\") pod \"340c0480-49d8-433b-90b0-194dd10ef44f\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") "
Sep 17 17:44:00.208315 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.208246    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-bundle\") pod \"340c0480-49d8-433b-90b0-194dd10ef44f\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") "
Sep 17 17:44:00.208315 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.208281    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-util\") pod \"340c0480-49d8-433b-90b0-194dd10ef44f\" (UID: \"340c0480-49d8-433b-90b0-194dd10ef44f\") "
Sep 17 17:44:00.208720 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.208690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-bundle" (OuterVolumeSpecName: "bundle") pod "340c0480-49d8-433b-90b0-194dd10ef44f" (UID: "340c0480-49d8-433b-90b0-194dd10ef44f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:00.210109 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.210088    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340c0480-49d8-433b-90b0-194dd10ef44f-kube-api-access-9x8xv" (OuterVolumeSpecName: "kube-api-access-9x8xv") pod "340c0480-49d8-433b-90b0-194dd10ef44f" (UID: "340c0480-49d8-433b-90b0-194dd10ef44f"). InnerVolumeSpecName "kube-api-access-9x8xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:00.213677 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.213658    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-util" (OuterVolumeSpecName: "util") pod "340c0480-49d8-433b-90b0-194dd10ef44f" (UID: "340c0480-49d8-433b-90b0-194dd10ef44f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:00.309089 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.309067    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x8xv\" (UniqueName: \"kubernetes.io/projected/340c0480-49d8-433b-90b0-194dd10ef44f-kube-api-access-9x8xv\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:00.309089 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.309089    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:00.309202 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.309099    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/340c0480-49d8-433b-90b0-194dd10ef44f-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:00.962483 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.962452    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerID="4a985e101eaaf875ac7301f6b4380c719a5fa05cf8214957b1bd97d03630a41d" exitCode=0
Sep 17 17:44:00.962861 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.962541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p" event={"ID":"6ba3c671-4fdd-4fd9-93dc-57ce544748db","Type":"ContainerDied","Data":"4a985e101eaaf875ac7301f6b4380c719a5fa05cf8214957b1bd97d03630a41d"}
Sep 17 17:44:00.964870 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.964849    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd"
Sep 17 17:44:00.964984 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.964865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pq5qd" event={"ID":"340c0480-49d8-433b-90b0-194dd10ef44f","Type":"ContainerDied","Data":"22147903ba5fae12e549222f093bb3037078e3e40021b34e2ad26e6904ca2392"}
Sep 17 17:44:00.964984 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.964905    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22147903ba5fae12e549222f093bb3037078e3e40021b34e2ad26e6904ca2392"
Sep 17 17:44:00.967568 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.967535    2581 generic.go:356] "Generic (PLEG): container finished" podID="ea71400a-4396-498b-9486-f543ac037955" containerID="cb4b8f2c13fc6191612e4f5005c2a18c40e04357f151749ad1a63d563c0a84b2" exitCode=0
Sep 17 17:44:00.967700 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:00.967568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9" event={"ID":"ea71400a-4396-498b-9486-f543ac037955","Type":"ContainerDied","Data":"cb4b8f2c13fc6191612e4f5005c2a18c40e04357f151749ad1a63d563c0a84b2"}
Sep 17 17:44:01.097617 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.097594    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:44:01.216396 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.216337    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-util\") pod \"68fb73d7-97b6-48da-bd21-32c181dc9a61\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") "
Sep 17 17:44:01.216501 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.216418    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwzf4\" (UniqueName: \"kubernetes.io/projected/68fb73d7-97b6-48da-bd21-32c181dc9a61-kube-api-access-dwzf4\") pod \"68fb73d7-97b6-48da-bd21-32c181dc9a61\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") "
Sep 17 17:44:01.216501 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.216466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-bundle\") pod \"68fb73d7-97b6-48da-bd21-32c181dc9a61\" (UID: \"68fb73d7-97b6-48da-bd21-32c181dc9a61\") "
Sep 17 17:44:01.216901 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.216874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-bundle" (OuterVolumeSpecName: "bundle") pod "68fb73d7-97b6-48da-bd21-32c181dc9a61" (UID: "68fb73d7-97b6-48da-bd21-32c181dc9a61"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:01.218434 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.218409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fb73d7-97b6-48da-bd21-32c181dc9a61-kube-api-access-dwzf4" (OuterVolumeSpecName: "kube-api-access-dwzf4") pod "68fb73d7-97b6-48da-bd21-32c181dc9a61" (UID: "68fb73d7-97b6-48da-bd21-32c181dc9a61"). InnerVolumeSpecName "kube-api-access-dwzf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:01.221729 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.221709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-util" (OuterVolumeSpecName: "util") pod "68fb73d7-97b6-48da-bd21-32c181dc9a61" (UID: "68fb73d7-97b6-48da-bd21-32c181dc9a61"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:01.317685 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.317659    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwzf4\" (UniqueName: \"kubernetes.io/projected/68fb73d7-97b6-48da-bd21-32c181dc9a61-kube-api-access-dwzf4\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:01.317685 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.317686    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:01.317860 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.317698    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68fb73d7-97b6-48da-bd21-32c181dc9a61-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:01.974920 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.974888    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl"
Sep 17 17:44:01.974920 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.974898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7drkjl" event={"ID":"68fb73d7-97b6-48da-bd21-32c181dc9a61","Type":"ContainerDied","Data":"bc720abefcc160646f856369f56c0fb615f27349a2110bf468da90de08716237"}
Sep 17 17:44:01.975333 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:01.974946    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc720abefcc160646f856369f56c0fb615f27349a2110bf468da90de08716237"
Sep 17 17:44:02.124478 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.124456    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:44:02.127553 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.127533    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:44:02.224494 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.224470    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-util\") pod \"ea71400a-4396-498b-9486-f543ac037955\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") "
Sep 17 17:44:02.224634 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.224502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-util\") pod \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") "
Sep 17 17:44:02.224634 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.224521    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr8hc\" (UniqueName: \"kubernetes.io/projected/6ba3c671-4fdd-4fd9-93dc-57ce544748db-kube-api-access-cr8hc\") pod \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") "
Sep 17 17:44:02.224634 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.224555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xqtv\" (UniqueName: \"kubernetes.io/projected/ea71400a-4396-498b-9486-f543ac037955-kube-api-access-9xqtv\") pod \"ea71400a-4396-498b-9486-f543ac037955\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") "
Sep 17 17:44:02.224790 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.224668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-bundle\") pod \"ea71400a-4396-498b-9486-f543ac037955\" (UID: \"ea71400a-4396-498b-9486-f543ac037955\") "
Sep 17 17:44:02.224790 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.224716    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-bundle\") pod \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\" (UID: \"6ba3c671-4fdd-4fd9-93dc-57ce544748db\") "
Sep 17 17:44:02.225386 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.225324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-bundle" (OuterVolumeSpecName: "bundle") pod "6ba3c671-4fdd-4fd9-93dc-57ce544748db" (UID: "6ba3c671-4fdd-4fd9-93dc-57ce544748db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:02.225489 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.225429    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-bundle" (OuterVolumeSpecName: "bundle") pod "ea71400a-4396-498b-9486-f543ac037955" (UID: "ea71400a-4396-498b-9486-f543ac037955"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:02.226689 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.226657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba3c671-4fdd-4fd9-93dc-57ce544748db-kube-api-access-cr8hc" (OuterVolumeSpecName: "kube-api-access-cr8hc") pod "6ba3c671-4fdd-4fd9-93dc-57ce544748db" (UID: "6ba3c671-4fdd-4fd9-93dc-57ce544748db"). InnerVolumeSpecName "kube-api-access-cr8hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:02.227158 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.227131    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea71400a-4396-498b-9486-f543ac037955-kube-api-access-9xqtv" (OuterVolumeSpecName: "kube-api-access-9xqtv") pod "ea71400a-4396-498b-9486-f543ac037955" (UID: "ea71400a-4396-498b-9486-f543ac037955"). InnerVolumeSpecName "kube-api-access-9xqtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:02.228600 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.228576    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-util" (OuterVolumeSpecName: "util") pod "6ba3c671-4fdd-4fd9-93dc-57ce544748db" (UID: "6ba3c671-4fdd-4fd9-93dc-57ce544748db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:02.230110 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.230087    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-util" (OuterVolumeSpecName: "util") pod "ea71400a-4396-498b-9486-f543ac037955" (UID: "ea71400a-4396-498b-9486-f543ac037955"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:02.325616 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.325583    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqtv\" (UniqueName: \"kubernetes.io/projected/ea71400a-4396-498b-9486-f543ac037955-kube-api-access-9xqtv\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:02.325616 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.325615    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:02.325616 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.325625    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:02.325763 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.325632    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea71400a-4396-498b-9486-f543ac037955-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:02.325763 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.325639    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ba3c671-4fdd-4fd9-93dc-57ce544748db-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:02.325763 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.325646    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr8hc\" (UniqueName: \"kubernetes.io/projected/6ba3c671-4fdd-4fd9-93dc-57ce544748db-kube-api-access-cr8hc\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:44:02.982565 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.982536    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p"
Sep 17 17:44:02.982948 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.982554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq699p" event={"ID":"6ba3c671-4fdd-4fd9-93dc-57ce544748db","Type":"ContainerDied","Data":"bcc2e5644eae27484768ed2b7027684d9fc1242ac4f368f5f87f0ce47e1ae5a8"}
Sep 17 17:44:02.982948 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.982609    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcc2e5644eae27484768ed2b7027684d9fc1242ac4f368f5f87f0ce47e1ae5a8"
Sep 17 17:44:02.984787 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.984769    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9"
Sep 17 17:44:02.984880 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.984803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6c2q9" event={"ID":"ea71400a-4396-498b-9486-f543ac037955","Type":"ContainerDied","Data":"059b4b0fe1429e12923bfa3987e84e2ad7d6a0673c4d58f349e187c71fbe768a"}
Sep 17 17:44:02.984880 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:02.984850    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="059b4b0fe1429e12923bfa3987e84e2ad7d6a0673c4d58f349e187c71fbe768a"
Sep 17 17:44:06.009667 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.009641    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:06.073553 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073521    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"]
Sep 17 17:44:06.073893 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073880    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerName="util"
Sep 17 17:44:06.073951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073896    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerName="util"
Sep 17 17:44:06.073951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073908    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea71400a-4396-498b-9486-f543ac037955" containerName="extract"
Sep 17 17:44:06.073951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073914    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea71400a-4396-498b-9486-f543ac037955" containerName="extract"
Sep 17 17:44:06.073951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073920    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerName="util"
Sep 17 17:44:06.073951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073927    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerName="util"
Sep 17 17:44:06.073951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073941    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerName="pull"
Sep 17 17:44:06.073951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073946    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerName="pull"
Sep 17 17:44:06.073951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073953    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerName="pull"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073958    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerName="pull"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073981    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerName="extract"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073987    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerName="extract"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.073995    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340c0480-49d8-433b-90b0-194dd10ef44f" containerName="extract"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074001    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="340c0480-49d8-433b-90b0-194dd10ef44f" containerName="extract"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074007    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerName="extract"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074013    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerName="extract"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074019    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340c0480-49d8-433b-90b0-194dd10ef44f" containerName="pull"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074024    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="340c0480-49d8-433b-90b0-194dd10ef44f" containerName="pull"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074031    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea71400a-4396-498b-9486-f543ac037955" containerName="util"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074036    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea71400a-4396-498b-9486-f543ac037955" containerName="util"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074042    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340c0480-49d8-433b-90b0-194dd10ef44f" containerName="util"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074049    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="340c0480-49d8-433b-90b0-194dd10ef44f" containerName="util"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074057    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea71400a-4396-498b-9486-f543ac037955" containerName="pull"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074064    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea71400a-4396-498b-9486-f543ac037955" containerName="pull"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074146    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba3c671-4fdd-4fd9-93dc-57ce544748db" containerName="extract"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074159    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="340c0480-49d8-433b-90b0-194dd10ef44f" containerName="extract"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074166    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="68fb73d7-97b6-48da-bd21-32c181dc9a61" containerName="extract"
Sep 17 17:44:06.074210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.074173    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea71400a-4396-498b-9486-f543ac037955" containerName="extract"
Sep 17 17:44:06.080896 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.080878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.085102 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.085082    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 17:44:06.085195 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.085105    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:44:06.085195 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.085163    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6n2s5\""
Sep 17 17:44:06.085195 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.085167    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 17:44:06.085344 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.085327    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 17:44:06.085525 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.085491    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 17:44:06.089286 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.089265    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"]
Sep 17 17:44:06.113766 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.113737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:06.154692 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.154670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f967781f-96ca-4b8c-abed-3ddfb3616ddb-manager-config\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.154797 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.154717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f967781f-96ca-4b8c-abed-3ddfb3616ddb-cert\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.154797 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.154738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25xgg\" (UniqueName: \"kubernetes.io/projected/f967781f-96ca-4b8c-abed-3ddfb3616ddb-kube-api-access-25xgg\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.154868 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.154795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f967781f-96ca-4b8c-abed-3ddfb3616ddb-metrics-cert\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.255585 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.255552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f967781f-96ca-4b8c-abed-3ddfb3616ddb-metrics-cert\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.255691 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.255610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f967781f-96ca-4b8c-abed-3ddfb3616ddb-manager-config\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.255691 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.255637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f967781f-96ca-4b8c-abed-3ddfb3616ddb-cert\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.255691 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.255655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xgg\" (UniqueName: \"kubernetes.io/projected/f967781f-96ca-4b8c-abed-3ddfb3616ddb-kube-api-access-25xgg\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.256365 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.256342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f967781f-96ca-4b8c-abed-3ddfb3616ddb-manager-config\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.258062 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.258041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f967781f-96ca-4b8c-abed-3ddfb3616ddb-metrics-cert\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.258147 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.258080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f967781f-96ca-4b8c-abed-3ddfb3616ddb-cert\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.267278 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.267226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25xgg\" (UniqueName: \"kubernetes.io/projected/f967781f-96ca-4b8c-abed-3ddfb3616ddb-kube-api-access-25xgg\") pod \"lws-controller-manager-ff6d497d7-fj4pr\" (UID: \"f967781f-96ca-4b8c-abed-3ddfb3616ddb\") " pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.390385 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.390363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:06.666850 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:06.666824    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"]
Sep 17 17:44:06.668707 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:44:06.668682    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf967781f_96ca_4b8c_abed_3ddfb3616ddb.slice/crio-bee47b87360a2d7bdb861a9012aef810b6369bbb51ebf2dffb4f216b62348af5 WatchSource:0}: Error finding container bee47b87360a2d7bdb861a9012aef810b6369bbb51ebf2dffb4f216b62348af5: Status 404 returned error can't find the container with id bee47b87360a2d7bdb861a9012aef810b6369bbb51ebf2dffb4f216b62348af5
Sep 17 17:44:07.005895 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:07.005862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr" event={"ID":"f967781f-96ca-4b8c-abed-3ddfb3616ddb","Type":"ContainerStarted","Data":"bee47b87360a2d7bdb861a9012aef810b6369bbb51ebf2dffb4f216b62348af5"}
Sep 17 17:44:07.606884 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:07.606851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:10.023836 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:10.023793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr" event={"ID":"f967781f-96ca-4b8c-abed-3ddfb3616ddb","Type":"ContainerStarted","Data":"2a7445207c2cece3082ae4c48694957498ab1cbc516788471ed2403845952757"}
Sep 17 17:44:10.024184 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:10.023862    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:10.047096 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:10.047013    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr" podStartSLOduration=0.933516775 podStartE2EDuration="4.046997819s" totalImagesPullingTime="3.113481044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:10.045713165 +0000 UTC m=+612.970185122" watchObservedRunningTime="2026-09-17 17:44:10.046997819 +0000 UTC m=+612.971469776"
Sep 17 17:44:10.438071 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:10.437960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:12.209709 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:12.209676    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:12.612288 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:12.612207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:12.897741 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:12.897664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:14.257208 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:14.257170    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:15.199723 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:15.199687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:15.280647 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:15.280612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:15.609180 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:15.609106    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:15.718617 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:15.718592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:16.032816 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:16.032781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:16.130127 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:16.130097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:16.841610 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:16.841581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:20.440898 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:20.440863    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:20.553047 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:20.553011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:21.031416 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:21.031389    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-ff6d497d7-fj4pr"
Sep 17 17:44:23.416795 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:23.416751    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:23.518246 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:23.518216    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:26.820524 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.820492    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"]
Sep 17 17:44:26.823850 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.823832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:26.827488 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.827464    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 17:44:26.827612 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.827492    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 17:44:26.828794 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.828745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 17:44:26.828794 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.828772    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wrlsg\""
Sep 17 17:44:26.828794 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.828777    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 17:44:26.839408 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.839386    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"]
Sep 17 17:44:26.845414 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.845392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:26.913927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.913901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56gb2\" (UniqueName: \"kubernetes.io/projected/dde2aab7-9294-4560-bf14-fb580f4f1268-kube-api-access-56gb2\") pod \"kuadrant-console-plugin-fbdd4f445-9bw7q\" (UID: \"dde2aab7-9294-4560-bf14-fb580f4f1268\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:26.914029 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.913940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dde2aab7-9294-4560-bf14-fb580f4f1268-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9bw7q\" (UID: \"dde2aab7-9294-4560-bf14-fb580f4f1268\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:26.914029 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.913977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dde2aab7-9294-4560-bf14-fb580f4f1268-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9bw7q\" (UID: \"dde2aab7-9294-4560-bf14-fb580f4f1268\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:26.928056 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:26.928032    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:27.014634 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:27.014609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56gb2\" (UniqueName: \"kubernetes.io/projected/dde2aab7-9294-4560-bf14-fb580f4f1268-kube-api-access-56gb2\") pod \"kuadrant-console-plugin-fbdd4f445-9bw7q\" (UID: \"dde2aab7-9294-4560-bf14-fb580f4f1268\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:27.014734 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:27.014642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dde2aab7-9294-4560-bf14-fb580f4f1268-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9bw7q\" (UID: \"dde2aab7-9294-4560-bf14-fb580f4f1268\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:27.014734 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:27.014659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dde2aab7-9294-4560-bf14-fb580f4f1268-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9bw7q\" (UID: \"dde2aab7-9294-4560-bf14-fb580f4f1268\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:27.015294 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:27.015278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dde2aab7-9294-4560-bf14-fb580f4f1268-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9bw7q\" (UID: \"dde2aab7-9294-4560-bf14-fb580f4f1268\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:27.017035 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:27.017017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dde2aab7-9294-4560-bf14-fb580f4f1268-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9bw7q\" (UID: \"dde2aab7-9294-4560-bf14-fb580f4f1268\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:27.023787 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:27.023768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56gb2\" (UniqueName: \"kubernetes.io/projected/dde2aab7-9294-4560-bf14-fb580f4f1268-kube-api-access-56gb2\") pod \"kuadrant-console-plugin-fbdd4f445-9bw7q\" (UID: \"dde2aab7-9294-4560-bf14-fb580f4f1268\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:27.132733 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:27.132684    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"
Sep 17 17:44:27.405896 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:27.405872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q"]
Sep 17 17:44:27.407121 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:44:27.407093    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddde2aab7_9294_4560_bf14_fb580f4f1268.slice/crio-8129ff46c9ded40fb447fd0afc3c012af87da795faec440ae271a19af7863fa6 WatchSource:0}: Error finding container 8129ff46c9ded40fb447fd0afc3c012af87da795faec440ae271a19af7863fa6: Status 404 returned error can't find the container with id 8129ff46c9ded40fb447fd0afc3c012af87da795faec440ae271a19af7863fa6
Sep 17 17:44:28.126249 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:28.126209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q" event={"ID":"dde2aab7-9294-4560-bf14-fb580f4f1268","Type":"ContainerStarted","Data":"8129ff46c9ded40fb447fd0afc3c012af87da795faec440ae271a19af7863fa6"}
Sep 17 17:44:29.042035 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:29.041997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:29.147237 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:29.147204    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:29.240127 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:29.240093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:33.163264 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:33.163220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q" event={"ID":"dde2aab7-9294-4560-bf14-fb580f4f1268","Type":"ContainerStarted","Data":"7e9881cbfc974c3891181d18fb278ac03b84141455a79bf51325232774f2a0db"}
Sep 17 17:44:33.185863 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:33.185807    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9bw7q" podStartSLOduration=2.347125848 podStartE2EDuration="7.18578913s" totalImagesPullingTime="4.838663282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:33.185391336 +0000 UTC m=+636.109863294" watchObservedRunningTime="2026-09-17 17:44:33.18578913 +0000 UTC m=+636.110261089"
Sep 17 17:44:33.847223 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:33.847185    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:33.937931 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:33.937890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:34.041266 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:34.041237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:37.030049 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:37.030010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:37.169251 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:37.169224    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:40.521767 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:40.521731    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:40.625153 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:40.625098    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:43.315839 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:43.315805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:43.419772 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:43.419731    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:48.117228 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:48.117190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:48.202343 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:48.202287    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:48.295810 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:48.295766    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:48.375798 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:48.375724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.262851 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:53.262816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.362734 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:53.362701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.468093 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:53.468071    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:57.049621 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:44:57.049576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:03.577479 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:03.577436    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:03.676645 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:03.676621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:10.801568 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:10.801533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:10.897016 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:10.896985    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:10.993731 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:10.993699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:11.247788 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.247763    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dpc59"]
Sep 17 17:45:11.265580 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.265554    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dpc59"]
Sep 17 17:45:11.265728 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.265651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:11.268427 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.268406    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 17:45:11.282015 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.281994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dpc59"]
Sep 17 17:45:11.351118 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.351092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7f5x\" (UniqueName: \"kubernetes.io/projected/a87d7776-a29c-4440-98cc-22b7fcfd8b31-kube-api-access-g7f5x\") pod \"limitador-limitador-84994f7ffc-dpc59\" (UID: \"a87d7776-a29c-4440-98cc-22b7fcfd8b31\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:11.351219 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.351131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a87d7776-a29c-4440-98cc-22b7fcfd8b31-config-file\") pod \"limitador-limitador-84994f7ffc-dpc59\" (UID: \"a87d7776-a29c-4440-98cc-22b7fcfd8b31\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:11.452417 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.452392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7f5x\" (UniqueName: \"kubernetes.io/projected/a87d7776-a29c-4440-98cc-22b7fcfd8b31-kube-api-access-g7f5x\") pod \"limitador-limitador-84994f7ffc-dpc59\" (UID: \"a87d7776-a29c-4440-98cc-22b7fcfd8b31\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:11.452534 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.452431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a87d7776-a29c-4440-98cc-22b7fcfd8b31-config-file\") pod \"limitador-limitador-84994f7ffc-dpc59\" (UID: \"a87d7776-a29c-4440-98cc-22b7fcfd8b31\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:11.452961 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.452942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a87d7776-a29c-4440-98cc-22b7fcfd8b31-config-file\") pod \"limitador-limitador-84994f7ffc-dpc59\" (UID: \"a87d7776-a29c-4440-98cc-22b7fcfd8b31\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:11.466789 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.466763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7f5x\" (UniqueName: \"kubernetes.io/projected/a87d7776-a29c-4440-98cc-22b7fcfd8b31-kube-api-access-g7f5x\") pod \"limitador-limitador-84994f7ffc-dpc59\" (UID: \"a87d7776-a29c-4440-98cc-22b7fcfd8b31\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:11.576176 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.576126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:11.661536 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.661497    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mfvw8"]
Sep 17 17:45:11.664510 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.664495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8"
Sep 17 17:45:11.666928 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.666911    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9htfg\""
Sep 17 17:45:11.680951 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.680930    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mfvw8"]
Sep 17 17:45:11.698169 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.698149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:11.755442 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.755416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44cm\" (UniqueName: \"kubernetes.io/projected/645371b0-61d0-4392-8998-545032973728-kube-api-access-z44cm\") pod \"authorino-7bcfd8f9cd-mfvw8\" (UID: \"645371b0-61d0-4392-8998-545032973728\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8"
Sep 17 17:45:11.796852 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.796833    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:11.845098 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.845047    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dpc59"]
Sep 17 17:45:11.848253 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:45:11.848227    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87d7776_a29c_4440_98cc_22b7fcfd8b31.slice/crio-3885c06f1c38d38d015108af7cf1abf2722baa25f35465fd87c54ae91ee428d8 WatchSource:0}: Error finding container 3885c06f1c38d38d015108af7cf1abf2722baa25f35465fd87c54ae91ee428d8: Status 404 returned error can't find the container with id 3885c06f1c38d38d015108af7cf1abf2722baa25f35465fd87c54ae91ee428d8
Sep 17 17:45:11.856750 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.856730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44cm\" (UniqueName: \"kubernetes.io/projected/645371b0-61d0-4392-8998-545032973728-kube-api-access-z44cm\") pod \"authorino-7bcfd8f9cd-mfvw8\" (UID: \"645371b0-61d0-4392-8998-545032973728\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8"
Sep 17 17:45:11.867008 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.866985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44cm\" (UniqueName: \"kubernetes.io/projected/645371b0-61d0-4392-8998-545032973728-kube-api-access-z44cm\") pod \"authorino-7bcfd8f9cd-mfvw8\" (UID: \"645371b0-61d0-4392-8998-545032973728\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8"
Sep 17 17:45:11.981884 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:11.981865    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8"
Sep 17 17:45:12.249846 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:12.249817    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mfvw8"]
Sep 17 17:45:12.251516 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:45:12.251486    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod645371b0_61d0_4392_8998_545032973728.slice/crio-55e8d789005994be3e91222e9c2532dba30c494039df23c4dd06182bc5714f13 WatchSource:0}: Error finding container 55e8d789005994be3e91222e9c2532dba30c494039df23c4dd06182bc5714f13: Status 404 returned error can't find the container with id 55e8d789005994be3e91222e9c2532dba30c494039df23c4dd06182bc5714f13
Sep 17 17:45:12.367038 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:12.367008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59" event={"ID":"a87d7776-a29c-4440-98cc-22b7fcfd8b31","Type":"ContainerStarted","Data":"3885c06f1c38d38d015108af7cf1abf2722baa25f35465fd87c54ae91ee428d8"}
Sep 17 17:45:12.368303 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:12.368278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8" event={"ID":"645371b0-61d0-4392-8998-545032973728","Type":"ContainerStarted","Data":"55e8d789005994be3e91222e9c2532dba30c494039df23c4dd06182bc5714f13"}
Sep 17 17:45:14.380447 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:14.380414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8" event={"ID":"645371b0-61d0-4392-8998-545032973728","Type":"ContainerStarted","Data":"0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16"}
Sep 17 17:45:14.395807 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:14.395751    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8" podStartSLOduration=1.906911027 podStartE2EDuration="3.395735024s" totalImagesPullingTime="1.488823997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:14.395036612 +0000 UTC m=+677.319508571" watchObservedRunningTime="2026-09-17 17:45:14.395735024 +0000 UTC m=+677.320206982"
Sep 17 17:45:15.160947 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:15.160913    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mfvw8"]
Sep 17 17:45:15.387231 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:15.387199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59" event={"ID":"a87d7776-a29c-4440-98cc-22b7fcfd8b31","Type":"ContainerStarted","Data":"9e669ce65be005ced7e5f7c8f2d4a1ccae21c5ab1782f6d030483174a9909227"}
Sep 17 17:45:15.406281 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:15.406226    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59" podStartSLOduration=1.676645129 podStartE2EDuration="4.406209362s" totalImagesPullingTime="2.729564233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:15.405117341 +0000 UTC m=+678.329589298" watchObservedRunningTime="2026-09-17 17:45:15.406209362 +0000 UTC m=+678.330681319"
Sep 17 17:45:16.392714 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:16.392674    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8" podUID="645371b0-61d0-4392-8998-545032973728" containerName="authorino" containerID="cri-o://0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16" gracePeriod=30
Sep 17 17:45:16.393139 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:16.392920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:16.632002 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:16.631960    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8"
Sep 17 17:45:16.697920 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:16.697903    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z44cm\" (UniqueName: \"kubernetes.io/projected/645371b0-61d0-4392-8998-545032973728-kube-api-access-z44cm\") pod \"645371b0-61d0-4392-8998-545032973728\" (UID: \"645371b0-61d0-4392-8998-545032973728\") "
Sep 17 17:45:16.699925 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:16.699905    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645371b0-61d0-4392-8998-545032973728-kube-api-access-z44cm" (OuterVolumeSpecName: "kube-api-access-z44cm") pod "645371b0-61d0-4392-8998-545032973728" (UID: "645371b0-61d0-4392-8998-545032973728"). InnerVolumeSpecName "kube-api-access-z44cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:16.799155 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:16.799134    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z44cm\" (UniqueName: \"kubernetes.io/projected/645371b0-61d0-4392-8998-545032973728-kube-api-access-z44cm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:45:17.407977 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:17.407928    2581 generic.go:356] "Generic (PLEG): container finished" podID="645371b0-61d0-4392-8998-545032973728" containerID="0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16" exitCode=0
Sep 17 17:45:17.408358 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:17.408010    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8"
Sep 17 17:45:17.408358 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:17.408017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8" event={"ID":"645371b0-61d0-4392-8998-545032973728","Type":"ContainerDied","Data":"0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16"}
Sep 17 17:45:17.408358 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:17.408070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mfvw8" event={"ID":"645371b0-61d0-4392-8998-545032973728","Type":"ContainerDied","Data":"55e8d789005994be3e91222e9c2532dba30c494039df23c4dd06182bc5714f13"}
Sep 17 17:45:17.408358 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:17.408089    2581 scope.go:117] "RemoveContainer" containerID="0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16"
Sep 17 17:45:17.417687 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:17.417665    2581 scope.go:117] "RemoveContainer" containerID="0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16"
Sep 17 17:45:17.417952 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:45:17.417929    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16\": container with ID starting with 0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16 not found: ID does not exist" containerID="0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16"
Sep 17 17:45:17.418040 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:17.417960    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16"} err="failed to get container status \"0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16\": rpc error: code = NotFound desc = could not find container \"0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16\": container with ID starting with 0eba8174f88026d4302a9e7ce73f709376d91101e6b4318b9c32eb0bfded2b16 not found: ID does not exist"
Sep 17 17:45:17.428903 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:17.428874    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mfvw8"]
Sep 17 17:45:17.436246 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:17.436222    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mfvw8"]
Sep 17 17:45:19.181741 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:19.181704    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="645371b0-61d0-4392-8998-545032973728" path="/var/lib/kubelet/pods/645371b0-61d0-4392-8998-545032973728/volumes"
Sep 17 17:45:27.409761 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:27.409723    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dpc59"
Sep 17 17:45:33.485112 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:33.485070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:33.894327 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:33.894250    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:35.957728 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:35.957681    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-7mxrx"]
Sep 17 17:45:35.958245 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:35.958090    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645371b0-61d0-4392-8998-545032973728" containerName="authorino"
Sep 17 17:45:35.958245 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:35.958106    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="645371b0-61d0-4392-8998-545032973728" containerName="authorino"
Sep 17 17:45:35.958245 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:35.958190    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="645371b0-61d0-4392-8998-545032973728" containerName="authorino"
Sep 17 17:45:35.962590 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:35.962570    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-7mxrx"
Sep 17 17:45:35.966358 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:35.966331    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 17:45:35.966358 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:35.966357    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9htfg\""
Sep 17 17:45:35.971025 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:35.970995    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-7mxrx"]
Sep 17 17:45:36.043895 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:36.043868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/069af643-5cf4-457e-9168-51b5e57908fc-tls-cert\") pod \"authorino-6d9898bd99-7mxrx\" (UID: \"069af643-5cf4-457e-9168-51b5e57908fc\") " pod="kuadrant-system/authorino-6d9898bd99-7mxrx"
Sep 17 17:45:36.044028 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:36.043915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxsx\" (UniqueName: \"kubernetes.io/projected/069af643-5cf4-457e-9168-51b5e57908fc-kube-api-access-8qxsx\") pod \"authorino-6d9898bd99-7mxrx\" (UID: \"069af643-5cf4-457e-9168-51b5e57908fc\") " pod="kuadrant-system/authorino-6d9898bd99-7mxrx"
Sep 17 17:45:36.144972 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:36.144945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/069af643-5cf4-457e-9168-51b5e57908fc-tls-cert\") pod \"authorino-6d9898bd99-7mxrx\" (UID: \"069af643-5cf4-457e-9168-51b5e57908fc\") " pod="kuadrant-system/authorino-6d9898bd99-7mxrx"
Sep 17 17:45:36.145101 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:36.145012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxsx\" (UniqueName: \"kubernetes.io/projected/069af643-5cf4-457e-9168-51b5e57908fc-kube-api-access-8qxsx\") pod \"authorino-6d9898bd99-7mxrx\" (UID: \"069af643-5cf4-457e-9168-51b5e57908fc\") " pod="kuadrant-system/authorino-6d9898bd99-7mxrx"
Sep 17 17:45:36.147877 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:36.147850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/069af643-5cf4-457e-9168-51b5e57908fc-tls-cert\") pod \"authorino-6d9898bd99-7mxrx\" (UID: \"069af643-5cf4-457e-9168-51b5e57908fc\") " pod="kuadrant-system/authorino-6d9898bd99-7mxrx"
Sep 17 17:45:36.159080 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:36.159058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxsx\" (UniqueName: \"kubernetes.io/projected/069af643-5cf4-457e-9168-51b5e57908fc-kube-api-access-8qxsx\") pod \"authorino-6d9898bd99-7mxrx\" (UID: \"069af643-5cf4-457e-9168-51b5e57908fc\") " pod="kuadrant-system/authorino-6d9898bd99-7mxrx"
Sep 17 17:45:36.272901 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:36.272852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-7mxrx"
Sep 17 17:45:36.547194 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:36.547168    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-7mxrx"]
Sep 17 17:45:36.548929 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:45:36.548900    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod069af643_5cf4_457e_9168_51b5e57908fc.slice/crio-21b2cf10bcfe7ec3b64f9005096274ab38581477d34f3d9436ed5394b5c8af99 WatchSource:0}: Error finding container 21b2cf10bcfe7ec3b64f9005096274ab38581477d34f3d9436ed5394b5c8af99: Status 404 returned error can't find the container with id 21b2cf10bcfe7ec3b64f9005096274ab38581477d34f3d9436ed5394b5c8af99
Sep 17 17:45:36.977484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:36.977448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:37.521262 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:37.521222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-7mxrx" event={"ID":"069af643-5cf4-457e-9168-51b5e57908fc","Type":"ContainerStarted","Data":"16a3028c895198261eb18a1b71715790e4cd38fe8813ce7b49809daae901738d"}
Sep 17 17:45:37.521452 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:37.521272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-7mxrx" event={"ID":"069af643-5cf4-457e-9168-51b5e57908fc","Type":"ContainerStarted","Data":"21b2cf10bcfe7ec3b64f9005096274ab38581477d34f3d9436ed5394b5c8af99"}
Sep 17 17:45:37.538934 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:37.538881    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-7mxrx" podStartSLOduration=1.991630242 podStartE2EDuration="2.538867297s" totalImagesPullingTime="547.237055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:37.537048736 +0000 UTC m=+700.461520694" watchObservedRunningTime="2026-09-17 17:45:37.538867297 +0000 UTC m=+700.463339254"
Sep 17 17:45:41.248697 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:41.248593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:41.393002 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:41.392950    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:41.492956 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:41.492913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:47.352795 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:45:47.352759    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:06.197482 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:46:06.197441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:06.265770 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:46:06.265743    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:06.374212 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:46:06.374185    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:06.498272 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:46:06.498196    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:06.789035 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:46:06.788926    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:06.864249 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:46:06.864227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:06.933089 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:46:06.933062    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:07.005385 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:46:07.005349    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:24.780445 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:47:24.780364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:24.870911 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:47:24.870882    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:25.014494 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:47:25.014465    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:25.123148 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:47:25.123075    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:25.215516 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:47:25.215495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:25.855032 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:47:25.855000    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:25.918140 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:47:25.918114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:46.245417 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:47:46.245376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:59.169121 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:48:59.169094    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:48:59.171652 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:48:59.171632    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:49:02.462595 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:02.462561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:02.563234 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:02.563202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:02.879363 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:02.879274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:02.984796 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:02.984763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:14.582730 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:14.582698    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:14.684074 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:14.684043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:24.359530 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:24.359478    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:24.792919 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:24.792886    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:34.801848 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:34.801805    2581 generic.go:356] "Generic (PLEG): container finished" podID="67f34990-ab55-4d22-b575-ae36e8990569" containerID="9bdbcc8f96e18c6bd992cb0da0f67f284fe259d285bcf93235e091451ab577f8" exitCode=0
Sep 17 17:49:34.802269 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:34.801883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" event={"ID":"67f34990-ab55-4d22-b575-ae36e8990569","Type":"ContainerDied","Data":"9bdbcc8f96e18c6bd992cb0da0f67f284fe259d285bcf93235e091451ab577f8"}
Sep 17 17:49:34.802362 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:34.802340    2581 scope.go:117] "RemoveContainer" containerID="9bdbcc8f96e18c6bd992cb0da0f67f284fe259d285bcf93235e091451ab577f8"
Sep 17 17:49:34.803526 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:34.803489    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:49:34.803586 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:34.803571    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:49:34.803626 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:34.803608    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:49:35.809026 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:35.808992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" event={"ID":"67f34990-ab55-4d22-b575-ae36e8990569","Type":"ContainerStarted","Data":"b05b7f7758533d3a38cec5424d2b3357a9b14f5ff84d643cfd051a484b72b466"}
Sep 17 17:49:35.888352 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:35.888314    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:49:35.889503 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:35.889478    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 17:49:35.889616 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:35.889525    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" podUID="67f34990-ab55-4d22-b575-ae36e8990569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 17:49:36.888620 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:36.888588    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 17:49:36.889002 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:36.888639    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" podUID="67f34990-ab55-4d22-b575-ae36e8990569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 17:49:37.641239 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.641207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:37.863602 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.863573    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"]
Sep 17 17:49:37.866158 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.866139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:49:37.870579 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.870561    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 17:49:37.871489 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.871465    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-vfndm\""
Sep 17 17:49:37.871611 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.871587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-dwrjn\""
Sep 17 17:49:37.871725 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.871683    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 17:49:37.871795 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.871635    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 17:49:37.888818 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.888788    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 17:49:37.889073 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.888840    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" podUID="67f34990-ab55-4d22-b575-ae36e8990569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 17:49:37.889928 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.889909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:49:37.894174 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.894126    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"]
Sep 17 17:49:37.937825 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.937765    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:37.954708 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.954690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg\" (UID: \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:49:37.954821 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:37.954743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6w99\" (UniqueName: \"kubernetes.io/projected/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-kube-api-access-k6w99\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg\" (UID: \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:49:38.056194 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.056166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6w99\" (UniqueName: \"kubernetes.io/projected/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-kube-api-access-k6w99\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg\" (UID: \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:49:38.056309 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.056228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg\" (UID: \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:49:38.058533 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.058516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg\" (UID: \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:49:38.065472 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.065453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6w99\" (UniqueName: \"kubernetes.io/projected/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-kube-api-access-k6w99\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg\" (UID: \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:49:38.176474 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.176414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:49:38.463903 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.463880    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"]
Sep 17 17:49:38.466112 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:49:38.466085    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15adf2c4_72c4_4e3d_a8cd_3f50590b21dc.slice/crio-ad3fad18d5953b59a9e27bf268c9fa6bf055a813da6c8eb44966793c9cd0d0fc WatchSource:0}: Error finding container ad3fad18d5953b59a9e27bf268c9fa6bf055a813da6c8eb44966793c9cd0d0fc: Status 404 returned error can't find the container with id ad3fad18d5953b59a9e27bf268c9fa6bf055a813da6c8eb44966793c9cd0d0fc
Sep 17 17:49:38.468157 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.468138    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:49:38.831069 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.830955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" event={"ID":"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc","Type":"ContainerStarted","Data":"ad3fad18d5953b59a9e27bf268c9fa6bf055a813da6c8eb44966793c9cd0d0fc"}
Sep 17 17:49:38.889406 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.889295    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 17:49:38.889406 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:38.889346    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" podUID="67f34990-ab55-4d22-b575-ae36e8990569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 17:49:39.889157 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:39.889128    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 17:49:39.889345 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:39.889180    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg" podUID="67f34990-ab55-4d22-b575-ae36e8990569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 17:49:41.225494 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:41.225464    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:49:41.227302 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:41.227278    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg"
Sep 17 17:49:41.858467 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:41.858427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" event={"ID":"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc","Type":"ContainerStarted","Data":"0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f"}
Sep 17 17:49:41.879275 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:41.879230    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" podStartSLOduration=2.297452987 podStartE2EDuration="4.879216017s" totalImagesPullingTime="2.58176303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:41.876102357 +0000 UTC m=+944.800574330" watchObservedRunningTime="2026-09-17 17:49:41.879216017 +0000 UTC m=+944.803687974"
Sep 17 17:49:42.863653 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:42.863619    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:49:50.522946 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:50.522907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:52.633437 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:49:52.633399    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:07.796333 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:07.796275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:10.287569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:10.287471    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:13.870390 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:13.870349    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:15.196257 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:15.196228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:15.215448 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:15.215418    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"]
Sep 17 17:50:15.217010 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:15.216979    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:15.294497 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:15.294472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:18.251020 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:18.250956    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:18.314214 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:18.314186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:25.217328 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:25.217288    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:30.234850 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:30.234802    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main" containerID="cri-o://0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f" gracePeriod=45
Sep 17 17:50:35.217630 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:35.217575    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:45.217810 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:45.217773    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:55.217339 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:50:55.217300    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:51:00.479419 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:00.479392    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:51:00.580305 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:00.580245    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-tls-certs\") pod \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\" (UID: \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\") "
Sep 17 17:51:00.580305 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:00.580298    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6w99\" (UniqueName: \"kubernetes.io/projected/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-kube-api-access-k6w99\") pod \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\" (UID: \"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc\") "
Sep 17 17:51:00.582296 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:00.582270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" (UID: "15adf2c4-72c4-4e3d-a8cd-3f50590b21dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:00.582296 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:00.582266    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-kube-api-access-k6w99" (OuterVolumeSpecName: "kube-api-access-k6w99") pod "15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" (UID: "15adf2c4-72c4-4e3d-a8cd-3f50590b21dc"). InnerVolumeSpecName "kube-api-access-k6w99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:00.681724 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:00.681702    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:51:00.681724 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:00.681724    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6w99\" (UniqueName: \"kubernetes.io/projected/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc-kube-api-access-k6w99\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:51:01.290482 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:01.290450    2581 generic.go:356] "Generic (PLEG): container finished" podID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerID="0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f" exitCode=0
Sep 17 17:51:01.290627 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:01.290542    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"
Sep 17 17:51:01.290627 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:01.290549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" event={"ID":"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc","Type":"ContainerDied","Data":"0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f"}
Sep 17 17:51:01.290717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:01.290654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg" event={"ID":"15adf2c4-72c4-4e3d-a8cd-3f50590b21dc","Type":"ContainerDied","Data":"ad3fad18d5953b59a9e27bf268c9fa6bf055a813da6c8eb44966793c9cd0d0fc"}
Sep 17 17:51:01.290717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:01.290679    2581 scope.go:117] "RemoveContainer" containerID="0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f"
Sep 17 17:51:01.301168 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:01.301148    2581 scope.go:117] "RemoveContainer" containerID="0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f"
Sep 17 17:51:01.303840 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:51:01.303818    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f\": container with ID starting with 0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f not found: ID does not exist" containerID="0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f"
Sep 17 17:51:01.303945 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:01.303845    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f"} err="failed to get container status \"0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f\": rpc error: code = NotFound desc = could not find container \"0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f\": container with ID starting with 0779a4b379ddb398ac4e3f54d9af282634012ef70adac78f8004e63ae238033f not found: ID does not exist"
Sep 17 17:51:01.312153 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:01.312116    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"]
Sep 17 17:51:01.314740 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:01.314719    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5nllcg"]
Sep 17 17:51:03.181042 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:03.181008    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" path="/var/lib/kubelet/pods/15adf2c4-72c4-4e3d-a8cd-3f50590b21dc/volumes"
Sep 17 17:51:04.126198 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:04.126164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:05.936314 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:05.936278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:08.328329 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:08.328293    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:08.423165 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:08.423110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:11.191442 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:11.191401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:12.511253 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:12.511216    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:12.737108 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:12.737077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:15.967891 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:15.967851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:17.537818 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:17.537778    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:17.632029 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:17.631911    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:22.434543 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:22.434514    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:22.622772 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:22.622739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:22.746192 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:22.746163    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:26.758762 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.758732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:26.991372 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.991342    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"]
Sep 17 17:51:26.991703 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.991692    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main"
Sep 17 17:51:26.991753 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.991706    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main"
Sep 17 17:51:26.991787 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.991769    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="15adf2c4-72c4-4e3d-a8cd-3f50590b21dc" containerName="main"
Sep 17 17:51:26.993879 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.993863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:51:26.998295 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.998268    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 17:51:26.998295 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.998285    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-6gk6p\""
Sep 17 17:51:26.998503 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.998292    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 17:51:26.998503 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.998268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-ct45r\""
Sep 17 17:51:26.998620 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:26.998590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 17:51:27.008109 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.008090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"]
Sep 17 17:51:27.014571 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.014515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:27.081601 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.081578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g\" (UID: \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:51:27.081694 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.081604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqk2h\" (UniqueName: \"kubernetes.io/projected/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-kube-api-access-lqk2h\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g\" (UID: \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:51:27.182028 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.182006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g\" (UID: \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:51:27.182137 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.182032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqk2h\" (UniqueName: \"kubernetes.io/projected/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-kube-api-access-lqk2h\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g\" (UID: \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:51:27.184479 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.184458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g\" (UID: \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:51:27.194234 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.194206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqk2h\" (UniqueName: \"kubernetes.io/projected/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-kube-api-access-lqk2h\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g\" (UID: \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:51:27.304388 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.304332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:51:27.577832 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:27.577803    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"]
Sep 17 17:51:27.579623 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:51:27.579585    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb623d3f8_ac5c_48cd_9f9e_33162cf6ec50.slice/crio-489031567bfaf70f4adcfce6ef2df0bb4f816ef783019ed8647d3daa99782223 WatchSource:0}: Error finding container 489031567bfaf70f4adcfce6ef2df0bb4f816ef783019ed8647d3daa99782223: Status 404 returned error can't find the container with id 489031567bfaf70f4adcfce6ef2df0bb4f816ef783019ed8647d3daa99782223
Sep 17 17:51:28.445918 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:28.445889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" event={"ID":"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50","Type":"ContainerStarted","Data":"25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896"}
Sep 17 17:51:28.446275 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:28.445934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" event={"ID":"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50","Type":"ContainerStarted","Data":"489031567bfaf70f4adcfce6ef2df0bb4f816ef783019ed8647d3daa99782223"}
Sep 17 17:51:28.446275 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:28.445985    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:51:28.472625 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:28.472583    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" podStartSLOduration=2.47257049 podStartE2EDuration="2.47257049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:28.470934972 +0000 UTC m=+1051.395406931" watchObservedRunningTime="2026-09-17 17:51:28.47257049 +0000 UTC m=+1051.397042447"
Sep 17 17:51:46.847018 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:46.846904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:51.386806 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:51.386771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:51.492768 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:51.492724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:59.452367 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:51:59.452344    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:52:06.536120 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:06.536066    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"]
Sep 17 17:52:06.564980 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:06.564937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:09.452543 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:09.452506    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:16.397540 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:16.397500    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:16.468353 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:16.468323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:19.452080 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:19.452043    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:21.565178 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:21.565137    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerName="main" containerID="cri-o://25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896" gracePeriod=45
Sep 17 17:52:29.452061 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:29.452023    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:29.452521 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:29.452141    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:52:39.451861 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:39.451825    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:39.951687 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:39.951653    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:40.038943 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:40.038910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:46.267109 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:46.267072    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:49.452082 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:49.452042    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:52:51.033491 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.033464    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:51.256087 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.256052    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:51.809877 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.809853    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:52:51.920386 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.920302    2581 generic.go:356] "Generic (PLEG): container finished" podID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerID="25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896" exitCode=0
Sep 17 17:52:51.920547 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.920430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" event={"ID":"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50","Type":"ContainerDied","Data":"25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896"}
Sep 17 17:52:51.920547 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.920495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g" event={"ID":"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50","Type":"ContainerDied","Data":"489031567bfaf70f4adcfce6ef2df0bb4f816ef783019ed8647d3daa99782223"}
Sep 17 17:52:51.920547 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.920539    2581 scope.go:117] "RemoveContainer" containerID="25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896"
Sep 17 17:52:51.920991 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.920730    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"
Sep 17 17:52:51.925293 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.925014    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqk2h\" (UniqueName: \"kubernetes.io/projected/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-kube-api-access-lqk2h\") pod \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\" (UID: \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\") "
Sep 17 17:52:51.925293 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.925126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-tls-certs\") pod \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\" (UID: \"b623d3f8-ac5c-48cd-9f9e-33162cf6ec50\") "
Sep 17 17:52:51.927716 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.927675    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" (UID: "b623d3f8-ac5c-48cd-9f9e-33162cf6ec50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:51.927825 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.927767    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-kube-api-access-lqk2h" (OuterVolumeSpecName: "kube-api-access-lqk2h") pod "b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" (UID: "b623d3f8-ac5c-48cd-9f9e-33162cf6ec50"). InnerVolumeSpecName "kube-api-access-lqk2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:51.934032 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.933998    2581 scope.go:117] "RemoveContainer" containerID="25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896"
Sep 17 17:52:51.934561 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:52:51.934315    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896\": container with ID starting with 25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896 not found: ID does not exist" containerID="25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896"
Sep 17 17:52:51.934561 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:51.934351    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896"} err="failed to get container status \"25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896\": rpc error: code = NotFound desc = could not find container \"25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896\": container with ID starting with 25880a1483c97d7104750f671c89e6a60c4d94f08bdab94036cfc03a10f2d896 not found: ID does not exist"
Sep 17 17:52:52.026345 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:52.026310    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqk2h\" (UniqueName: \"kubernetes.io/projected/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-kube-api-access-lqk2h\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:52:52.026345 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:52.026333    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:52:52.249754 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:52.249714    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"]
Sep 17 17:52:52.252132 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:52.252106    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8588g"]
Sep 17 17:52:52.961188 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:52.961156    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:53.060921 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:53.060894    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:53.180934 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:53.180905    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" path="/var/lib/kubelet/pods/b623d3f8-ac5c-48cd-9f9e-33162cf6ec50/volumes"
Sep 17 17:52:56.419927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.419891    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:56.522166 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.522138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:56.620578 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.620555    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:56.644742 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.644717    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"]
Sep 17 17:52:56.645120 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.645105    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerName="main"
Sep 17 17:52:56.645171 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.645122    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerName="main"
Sep 17 17:52:56.645211 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.645191    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b623d3f8-ac5c-48cd-9f9e-33162cf6ec50" containerName="main"
Sep 17 17:52:56.648234 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.648214    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.651393 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.651368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-mgtv2\""
Sep 17 17:52:56.651494 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.651370    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:52:56.651494 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.651398    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 17:52:56.651494 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.651376    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 17:52:56.663038 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.663014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"]
Sep 17 17:52:56.720881 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.720855    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:56.765230 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.765205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.765327 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.765252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.765327 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.765312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26gng\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-kube-api-access-26gng\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.765414 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.765337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991efe21-924f-45c1-af34-1492ca995ea6-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.765414 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.765393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.765414 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.765408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.765525 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.765428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.765525 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.765464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.765525 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.765506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991efe21-924f-45c1-af34-1492ca995ea6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866159 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866262 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gng\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-kube-api-access-26gng\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866262 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991efe21-924f-45c1-af34-1492ca995ea6-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866262 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866262 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866461 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866461 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866461 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991efe21-924f-45c1-af34-1492ca995ea6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866461 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866533    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.866874 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.866796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.867178 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.867156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991efe21-924f-45c1-af34-1492ca995ea6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.868486 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.868465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.868677 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.868662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991efe21-924f-45c1-af34-1492ca995ea6-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.878802 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.878776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.879472 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.879452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gng\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-kube-api-access-26gng\") pod \"router-gateway-1-openshift-default-6b46477c5b-l9mm5\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:56.959872 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:56.959847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:57.236942 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:57.236917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"]
Sep 17 17:52:57.237809 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:52:57.237782    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991efe21_924f_45c1_af34_1492ca995ea6.slice/crio-673c15918133ad0a3bebcb8da6ab966216c7836571f4c024dea5d1de0b576c61 WatchSource:0}: Error finding container 673c15918133ad0a3bebcb8da6ab966216c7836571f4c024dea5d1de0b576c61: Status 404 returned error can't find the container with id 673c15918133ad0a3bebcb8da6ab966216c7836571f4c024dea5d1de0b576c61
Sep 17 17:52:57.239902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:57.239871    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:52:57.240002 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:57.239938    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:52:57.240002 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:57.239979    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 17:52:57.955745 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:57.955709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5" event={"ID":"991efe21-924f-45c1-af34-1492ca995ea6","Type":"ContainerStarted","Data":"5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a"}
Sep 17 17:52:57.956103 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:57.955800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5" event={"ID":"991efe21-924f-45c1-af34-1492ca995ea6","Type":"ContainerStarted","Data":"673c15918133ad0a3bebcb8da6ab966216c7836571f4c024dea5d1de0b576c61"}
Sep 17 17:52:57.959922 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:57.959901    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:57.979291 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:57.979249    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5" podStartSLOduration=1.979235395 podStartE2EDuration="1.979235395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:57.977279062 +0000 UTC m=+1140.901751019" watchObservedRunningTime="2026-09-17 17:52:57.979235395 +0000 UTC m=+1140.903707352"
Sep 17 17:52:58.965738 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:58.965696    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:59.967147 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:59.967115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:52:59.968336 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:52:59.968313    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:53:00.174374 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:00.174347    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:03.006157 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:03.006120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:04.507018 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:04.506962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:09.128463 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:09.128425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:09.846569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:09.846532    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:09.923315 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:09.923281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:25.119017 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.118904    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"]
Sep 17 17:53:25.122752 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.122726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.126328 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.126305    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-5f4vv\""
Sep 17 17:53:25.126328 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.126323    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 17:53:25.130623 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.130599    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"]
Sep 17 17:53:25.146092 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.146066    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:25.179651 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.179630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.179788 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.179659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.179788 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.179701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfl5p\" (UniqueName: \"kubernetes.io/projected/de813dbd-f085-4293-b8ac-bfe34f9a4c32-kube-api-access-sfl5p\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.179909 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.179793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.179909 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.179836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.179909 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.179853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.281036 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.281009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.281162 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.281058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.281162 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.281085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.281162 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.281149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.281327 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.281176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.281327 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.281231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfl5p\" (UniqueName: \"kubernetes.io/projected/de813dbd-f085-4293-b8ac-bfe34f9a4c32-kube-api-access-sfl5p\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.281457 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.281437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.281518 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.281500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.281584 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.281568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.283304 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.283276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.283559 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.283541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.290150 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.290122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfl5p\" (UniqueName: \"kubernetes.io/projected/de813dbd-f085-4293-b8ac-bfe34f9a4c32-kube-api-access-sfl5p\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.435635 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.435571    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:25.719872 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:25.719848    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"]
Sep 17 17:53:25.721281 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:53:25.721247    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde813dbd_f085_4293_b8ac_bfe34f9a4c32.slice/crio-2f4b40e7d9ff34bcb081bdb02cf0ee6e82165055e165f3a04d7b8068144b7caf WatchSource:0}: Error finding container 2f4b40e7d9ff34bcb081bdb02cf0ee6e82165055e165f3a04d7b8068144b7caf: Status 404 returned error can't find the container with id 2f4b40e7d9ff34bcb081bdb02cf0ee6e82165055e165f3a04d7b8068144b7caf
Sep 17 17:53:26.128934 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:26.128854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc" event={"ID":"de813dbd-f085-4293-b8ac-bfe34f9a4c32","Type":"ContainerStarted","Data":"2f4b40e7d9ff34bcb081bdb02cf0ee6e82165055e165f3a04d7b8068144b7caf"}
Sep 17 17:53:27.333487 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:27.333424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:27.437582 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:27.437512    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:27.534726 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:27.534693    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:27.660648 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:27.660618    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:27.742422 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:27.742387    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:28.148638 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:28.148606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc" event={"ID":"de813dbd-f085-4293-b8ac-bfe34f9a4c32","Type":"ContainerStarted","Data":"9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd"}
Sep 17 17:53:28.167167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:28.167121    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc" podStartSLOduration=1.403486531 podStartE2EDuration="3.167104115s" totalImagesPullingTime="1.763617584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:28.165808392 +0000 UTC m=+1171.090280350" watchObservedRunningTime="2026-09-17 17:53:28.167104115 +0000 UTC m=+1171.091576072"
Sep 17 17:53:28.436668 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:28.436581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:30.840647 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:30.840618    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:31.535876 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.535831    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"]
Sep 17 17:53:31.545323 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:53:31.545286    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:53:31.545323 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:53:31.545323 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:53:31.545323 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd" cmd=["/bin/sleep","15"]
Sep 17 17:53:31.545658 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.545341    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc" message="" err=<
Sep 17 17:53:31.545658 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:53:31.545658 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:53:31.545658 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 17 17:53:31.545658 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:53:31.545357    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:53:31.545658 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:53:31.545658 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:53:31.545658 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc" podUID="de813dbd-f085-4293-b8ac-bfe34f9a4c32" containerName="main" containerID="cri-o://9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd"
Sep 17 17:53:31.545658 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.545395    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc" podUID="de813dbd-f085-4293-b8ac-bfe34f9a4c32" containerName="main" containerID="cri-o://9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd" gracePeriod=60
Sep 17 17:53:31.578162 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.578129    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:31.795033 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.794978    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:31.847464 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.847442    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-home\") pod \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") "
Sep 17 17:53:31.847836 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.847484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tmp-dir\") pod \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") "
Sep 17 17:53:31.847836 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.847510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-dshm\") pod \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") "
Sep 17 17:53:31.847836 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.847546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tls-certs\") pod \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") "
Sep 17 17:53:31.847836 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.847650    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-model-cache\") pod \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") "
Sep 17 17:53:31.847836 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.847681    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfl5p\" (UniqueName: \"kubernetes.io/projected/de813dbd-f085-4293-b8ac-bfe34f9a4c32-kube-api-access-sfl5p\") pod \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\" (UID: \"de813dbd-f085-4293-b8ac-bfe34f9a4c32\") "
Sep 17 17:53:31.847836 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.847684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-home" (OuterVolumeSpecName: "home") pod "de813dbd-f085-4293-b8ac-bfe34f9a4c32" (UID: "de813dbd-f085-4293-b8ac-bfe34f9a4c32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:31.847836 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.847709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de813dbd-f085-4293-b8ac-bfe34f9a4c32" (UID: "de813dbd-f085-4293-b8ac-bfe34f9a4c32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:31.848179 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.847920    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-model-cache" (OuterVolumeSpecName: "model-cache") pod "de813dbd-f085-4293-b8ac-bfe34f9a4c32" (UID: "de813dbd-f085-4293-b8ac-bfe34f9a4c32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:31.848179 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.848044    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:31.848179 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.848066    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:31.848179 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.848079    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:31.849805 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.849779    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de813dbd-f085-4293-b8ac-bfe34f9a4c32-kube-api-access-sfl5p" (OuterVolumeSpecName: "kube-api-access-sfl5p") pod "de813dbd-f085-4293-b8ac-bfe34f9a4c32" (UID: "de813dbd-f085-4293-b8ac-bfe34f9a4c32"). InnerVolumeSpecName "kube-api-access-sfl5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:31.849917 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.849894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-dshm" (OuterVolumeSpecName: "dshm") pod "de813dbd-f085-4293-b8ac-bfe34f9a4c32" (UID: "de813dbd-f085-4293-b8ac-bfe34f9a4c32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:31.849917 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.849891    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de813dbd-f085-4293-b8ac-bfe34f9a4c32" (UID: "de813dbd-f085-4293-b8ac-bfe34f9a4c32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:31.949122 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.949101    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfl5p\" (UniqueName: \"kubernetes.io/projected/de813dbd-f085-4293-b8ac-bfe34f9a4c32-kube-api-access-sfl5p\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:31.949122 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.949119    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de813dbd-f085-4293-b8ac-bfe34f9a4c32-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:31.949253 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:31.949127    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de813dbd-f085-4293-b8ac-bfe34f9a4c32-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:32.170810 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.170733    2581 generic.go:356] "Generic (PLEG): container finished" podID="de813dbd-f085-4293-b8ac-bfe34f9a4c32" containerID="9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd" exitCode=0
Sep 17 17:53:32.170810 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.170798    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"
Sep 17 17:53:32.171070 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.170822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc" event={"ID":"de813dbd-f085-4293-b8ac-bfe34f9a4c32","Type":"ContainerDied","Data":"9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd"}
Sep 17 17:53:32.171070 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.170887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc" event={"ID":"de813dbd-f085-4293-b8ac-bfe34f9a4c32","Type":"ContainerDied","Data":"2f4b40e7d9ff34bcb081bdb02cf0ee6e82165055e165f3a04d7b8068144b7caf"}
Sep 17 17:53:32.171070 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.170918    2581 scope.go:117] "RemoveContainer" containerID="9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd"
Sep 17 17:53:32.180797 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.180778    2581 scope.go:117] "RemoveContainer" containerID="9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd"
Sep 17 17:53:32.181145 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:53:32.181118    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd\": container with ID starting with 9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd not found: ID does not exist" containerID="9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd"
Sep 17 17:53:32.181231 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.181153    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd"} err="failed to get container status \"9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd\": rpc error: code = NotFound desc = could not find container \"9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd\": container with ID starting with 9b9657aae1c4f73faabe91b0658d4b21a0eaa50a8110d21fcffab9127a2122cd not found: ID does not exist"
Sep 17 17:53:32.201593 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.201569    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"]
Sep 17 17:53:32.212557 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.212534    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-7rvxc"]
Sep 17 17:53:32.306242 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.306215    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"]
Sep 17 17:53:32.306473 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.306451    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5" podUID="991efe21-924f-45c1-af34-1492ca995ea6" containerName="istio-proxy" containerID="cri-o://5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a" gracePeriod=30
Sep 17 17:53:32.398774 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:32.398737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:33.186248 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:33.186213    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de813dbd-f085-4293-b8ac-bfe34f9a4c32" path="/var/lib/kubelet/pods/de813dbd-f085-4293-b8ac-bfe34f9a4c32/volumes"
Sep 17 17:53:34.748654 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.748631    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:53:34.872667 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872645    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-envoy\") pod \"991efe21-924f-45c1-af34-1492ca995ea6\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") "
Sep 17 17:53:34.872767 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872681    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-certs\") pod \"991efe21-924f-45c1-af34-1492ca995ea6\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") "
Sep 17 17:53:34.872767 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-socket\") pod \"991efe21-924f-45c1-af34-1492ca995ea6\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") "
Sep 17 17:53:34.872767 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872747    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991efe21-924f-45c1-af34-1492ca995ea6-istiod-ca-cert\") pod \"991efe21-924f-45c1-af34-1492ca995ea6\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") "
Sep 17 17:53:34.872909 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-data\") pod \"991efe21-924f-45c1-af34-1492ca995ea6\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") "
Sep 17 17:53:34.872909 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-credential-socket\") pod \"991efe21-924f-45c1-af34-1492ca995ea6\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") "
Sep 17 17:53:34.872909 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872889    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26gng\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-kube-api-access-26gng\") pod \"991efe21-924f-45c1-af34-1492ca995ea6\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") "
Sep 17 17:53:34.873081 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991efe21-924f-45c1-af34-1492ca995ea6-istio-podinfo\") pod \"991efe21-924f-45c1-af34-1492ca995ea6\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") "
Sep 17 17:53:34.873081 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-istio-token\") pod \"991efe21-924f-45c1-af34-1492ca995ea6\" (UID: \"991efe21-924f-45c1-af34-1492ca995ea6\") "
Sep 17 17:53:34.873081 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.872984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "991efe21-924f-45c1-af34-1492ca995ea6" (UID: "991efe21-924f-45c1-af34-1492ca995ea6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:34.873215 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.873087    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "991efe21-924f-45c1-af34-1492ca995ea6" (UID: "991efe21-924f-45c1-af34-1492ca995ea6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:34.873215 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.873110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-data" (OuterVolumeSpecName: "istio-data") pod "991efe21-924f-45c1-af34-1492ca995ea6" (UID: "991efe21-924f-45c1-af34-1492ca995ea6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:34.873215 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.873178    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991efe21-924f-45c1-af34-1492ca995ea6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "991efe21-924f-45c1-af34-1492ca995ea6" (UID: "991efe21-924f-45c1-af34-1492ca995ea6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:34.873350 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.873319    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:34.873350 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.873337    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-workload-socket\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:34.873454 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.873420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "991efe21-924f-45c1-af34-1492ca995ea6" (UID: "991efe21-924f-45c1-af34-1492ca995ea6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:34.873454 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.873448    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991efe21-924f-45c1-af34-1492ca995ea6-istiod-ca-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:34.873529 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.873465    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-data\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:34.874937 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.874912    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/991efe21-924f-45c1-af34-1492ca995ea6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "991efe21-924f-45c1-af34-1492ca995ea6" (UID: "991efe21-924f-45c1-af34-1492ca995ea6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 17:53:34.875059 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.874946    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-istio-token" (OuterVolumeSpecName: "istio-token") pod "991efe21-924f-45c1-af34-1492ca995ea6" (UID: "991efe21-924f-45c1-af34-1492ca995ea6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:34.875146 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.875130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "991efe21-924f-45c1-af34-1492ca995ea6" (UID: "991efe21-924f-45c1-af34-1492ca995ea6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:34.875203 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.875160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-kube-api-access-26gng" (OuterVolumeSpecName: "kube-api-access-26gng") pod "991efe21-924f-45c1-af34-1492ca995ea6" (UID: "991efe21-924f-45c1-af34-1492ca995ea6"). InnerVolumeSpecName "kube-api-access-26gng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:34.974357 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.974336    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-istio-token\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:34.974357 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.974356    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-istio-envoy\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:34.974498 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.974364    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991efe21-924f-45c1-af34-1492ca995ea6-credential-socket\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:34.974498 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.974373    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26gng\" (UniqueName: \"kubernetes.io/projected/991efe21-924f-45c1-af34-1492ca995ea6-kube-api-access-26gng\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:34.974498 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:34.974382    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991efe21-924f-45c1-af34-1492ca995ea6-istio-podinfo\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:35.188927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:35.188703    2581 generic.go:356] "Generic (PLEG): container finished" podID="991efe21-924f-45c1-af34-1492ca995ea6" containerID="5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a" exitCode=0
Sep 17 17:53:35.188927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:35.188769    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"
Sep 17 17:53:35.188927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:35.188777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5" event={"ID":"991efe21-924f-45c1-af34-1492ca995ea6","Type":"ContainerDied","Data":"5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a"}
Sep 17 17:53:35.188927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:35.188835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5" event={"ID":"991efe21-924f-45c1-af34-1492ca995ea6","Type":"ContainerDied","Data":"673c15918133ad0a3bebcb8da6ab966216c7836571f4c024dea5d1de0b576c61"}
Sep 17 17:53:35.188927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:35.188863    2581 scope.go:117] "RemoveContainer" containerID="5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a"
Sep 17 17:53:35.199251 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:35.199231    2581 scope.go:117] "RemoveContainer" containerID="5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a"
Sep 17 17:53:35.199567 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:53:35.199544    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a\": container with ID starting with 5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a not found: ID does not exist" containerID="5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a"
Sep 17 17:53:35.199642 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:35.199577    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a"} err="failed to get container status \"5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a\": rpc error: code = NotFound desc = could not find container \"5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a\": container with ID starting with 5505ef765ef76e8db434858e9dcac781ec6418327328fa328c40a9c42fb3241a not found: ID does not exist"
Sep 17 17:53:35.211556 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:35.211527    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"]
Sep 17 17:53:35.222033 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:35.222006    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l9mm5"]
Sep 17 17:53:37.181820 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.181789    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="991efe21-924f-45c1-af34-1492ca995ea6" path="/var/lib/kubelet/pods/991efe21-924f-45c1-af34-1492ca995ea6/volumes"
Sep 17 17:53:37.289114 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.289090    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"]
Sep 17 17:53:37.289498 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.289479    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991efe21-924f-45c1-af34-1492ca995ea6" containerName="istio-proxy"
Sep 17 17:53:37.289498 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.289496    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="991efe21-924f-45c1-af34-1492ca995ea6" containerName="istio-proxy"
Sep 17 17:53:37.289705 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.289508    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de813dbd-f085-4293-b8ac-bfe34f9a4c32" containerName="main"
Sep 17 17:53:37.289705 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.289513    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de813dbd-f085-4293-b8ac-bfe34f9a4c32" containerName="main"
Sep 17 17:53:37.289705 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.289599    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="991efe21-924f-45c1-af34-1492ca995ea6" containerName="istio-proxy"
Sep 17 17:53:37.289705 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.289615    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de813dbd-f085-4293-b8ac-bfe34f9a4c32" containerName="main"
Sep 17 17:53:37.292713 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.292693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.296661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.296595    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-5mbcf\""
Sep 17 17:53:37.296661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.296613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 17:53:37.296661 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.296595    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 17:53:37.296923 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.296629    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 17:53:37.305854 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.305833    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"]
Sep 17 17:53:37.332257 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.332233    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:37.393844 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.393821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.393936 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.393859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.393936 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.393898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3f724b6-4899-481c-aa7b-a0eb892de18f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.394088 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.393933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7s2n\" (UniqueName: \"kubernetes.io/projected/c3f724b6-4899-481c-aa7b-a0eb892de18f-kube-api-access-m7s2n\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.394088 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.394036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.394186 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.394103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.495250 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.495228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.495348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.495271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.495348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.495299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.495348 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.495323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.495481 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.495347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3f724b6-4899-481c-aa7b-a0eb892de18f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.495481 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.495376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7s2n\" (UniqueName: \"kubernetes.io/projected/c3f724b6-4899-481c-aa7b-a0eb892de18f-kube-api-access-m7s2n\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.495769 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.495733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.495769 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.495763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.495892 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.495785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.497444 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.497425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.497687 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.497669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3f724b6-4899-481c-aa7b-a0eb892de18f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.504505 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.504482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7s2n\" (UniqueName: \"kubernetes.io/projected/c3f724b6-4899-481c-aa7b-a0eb892de18f-kube-api-access-m7s2n\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.557935 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.557913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:37.604085 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.604061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:37.891921 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:37.891894    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"]
Sep 17 17:53:37.893241 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:53:37.893210    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f724b6_4899_481c_aa7b_a0eb892de18f.slice/crio-82b01b16d4ad5c4142cbcf110df5b8a0c53bafac978a9b56e604df1418039dd5 WatchSource:0}: Error finding container 82b01b16d4ad5c4142cbcf110df5b8a0c53bafac978a9b56e604df1418039dd5: Status 404 returned error can't find the container with id 82b01b16d4ad5c4142cbcf110df5b8a0c53bafac978a9b56e604df1418039dd5
Sep 17 17:53:38.206725 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:38.206689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h" event={"ID":"c3f724b6-4899-481c-aa7b-a0eb892de18f","Type":"ContainerStarted","Data":"b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c"}
Sep 17 17:53:38.207103 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:38.206780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h" event={"ID":"c3f724b6-4899-481c-aa7b-a0eb892de18f","Type":"ContainerStarted","Data":"82b01b16d4ad5c4142cbcf110df5b8a0c53bafac978a9b56e604df1418039dd5"}
Sep 17 17:53:38.234636 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:38.234592    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h" podStartSLOduration=1.23457712 podStartE2EDuration="1.23457712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:38.23326596 +0000 UTC m=+1181.157737931" watchObservedRunningTime="2026-09-17 17:53:38.23457712 +0000 UTC m=+1181.159049077"
Sep 17 17:53:38.605072 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:38.605008    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:40.259291 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:40.259226    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:40.357472 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:40.357445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:40.464926 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:40.464901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:41.155728 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:41.155700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:41.733470 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:41.733430    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"]
Sep 17 17:53:41.739688 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:41.739664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:41.742486 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:53:41.742456    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:53:41.742486 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:53:41.742486 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:53:41.742486 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c" cmd=["/bin/sleep","15"]
Sep 17 17:53:41.742684 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:41.742497    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h" message="" err=<
Sep 17 17:53:41.742684 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:53:41.742684 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:53:41.742684 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 17 17:53:41.742684 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:53:41.742511    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:53:41.742684 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:53:41.742684 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:53:41.742684 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h" podUID="c3f724b6-4899-481c-aa7b-a0eb892de18f" containerName="main" containerID="cri-o://b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c"
Sep 17 17:53:41.742684 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:41.742553    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h" podUID="c3f724b6-4899-481c-aa7b-a0eb892de18f" containerName="main" containerID="cri-o://b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c" gracePeriod=60
Sep 17 17:53:41.977317 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:41.977297    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:42.028556 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028497    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-home\") pod \"c3f724b6-4899-481c-aa7b-a0eb892de18f\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") "
Sep 17 17:53:42.028556 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-dshm\") pod \"c3f724b6-4899-481c-aa7b-a0eb892de18f\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") "
Sep 17 17:53:42.028714 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028594    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-tmp-dir\") pod \"c3f724b6-4899-481c-aa7b-a0eb892de18f\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") "
Sep 17 17:53:42.028714 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028616    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7s2n\" (UniqueName: \"kubernetes.io/projected/c3f724b6-4899-481c-aa7b-a0eb892de18f-kube-api-access-m7s2n\") pod \"c3f724b6-4899-481c-aa7b-a0eb892de18f\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") "
Sep 17 17:53:42.028714 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3f724b6-4899-481c-aa7b-a0eb892de18f-tls-certs\") pod \"c3f724b6-4899-481c-aa7b-a0eb892de18f\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") "
Sep 17 17:53:42.028714 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028655    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-model-cache\") pod \"c3f724b6-4899-481c-aa7b-a0eb892de18f\" (UID: \"c3f724b6-4899-481c-aa7b-a0eb892de18f\") "
Sep 17 17:53:42.028915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028744    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-home" (OuterVolumeSpecName: "home") pod "c3f724b6-4899-481c-aa7b-a0eb892de18f" (UID: "c3f724b6-4899-481c-aa7b-a0eb892de18f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:42.028915 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3f724b6-4899-481c-aa7b-a0eb892de18f" (UID: "c3f724b6-4899-481c-aa7b-a0eb892de18f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:42.029052 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3f724b6-4899-481c-aa7b-a0eb892de18f" (UID: "c3f724b6-4899-481c-aa7b-a0eb892de18f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:42.029052 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.028954    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:42.029052 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.029007    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:42.030725 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.030700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-dshm" (OuterVolumeSpecName: "dshm") pod "c3f724b6-4899-481c-aa7b-a0eb892de18f" (UID: "c3f724b6-4899-481c-aa7b-a0eb892de18f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:42.030725 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.030713    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3f724b6-4899-481c-aa7b-a0eb892de18f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3f724b6-4899-481c-aa7b-a0eb892de18f" (UID: "c3f724b6-4899-481c-aa7b-a0eb892de18f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:42.030851 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.030762    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f724b6-4899-481c-aa7b-a0eb892de18f-kube-api-access-m7s2n" (OuterVolumeSpecName: "kube-api-access-m7s2n") pod "c3f724b6-4899-481c-aa7b-a0eb892de18f" (UID: "c3f724b6-4899-481c-aa7b-a0eb892de18f"). InnerVolumeSpecName "kube-api-access-m7s2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:42.130196 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.130170    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:42.130196 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.130192    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7s2n\" (UniqueName: \"kubernetes.io/projected/c3f724b6-4899-481c-aa7b-a0eb892de18f-kube-api-access-m7s2n\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:42.130324 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.130208    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3f724b6-4899-481c-aa7b-a0eb892de18f-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:42.130324 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.130219    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3f724b6-4899-481c-aa7b-a0eb892de18f-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:53:42.229038 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.229011    2581 generic.go:356] "Generic (PLEG): container finished" podID="c3f724b6-4899-481c-aa7b-a0eb892de18f" containerID="b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c" exitCode=0
Sep 17 17:53:42.229128 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.229046    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"
Sep 17 17:53:42.229128 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.229096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h" event={"ID":"c3f724b6-4899-481c-aa7b-a0eb892de18f","Type":"ContainerDied","Data":"b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c"}
Sep 17 17:53:42.229203 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.229137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h" event={"ID":"c3f724b6-4899-481c-aa7b-a0eb892de18f","Type":"ContainerDied","Data":"82b01b16d4ad5c4142cbcf110df5b8a0c53bafac978a9b56e604df1418039dd5"}
Sep 17 17:53:42.229203 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.229159    2581 scope.go:117] "RemoveContainer" containerID="b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c"
Sep 17 17:53:42.238290 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.238276    2581 scope.go:117] "RemoveContainer" containerID="b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c"
Sep 17 17:53:42.238574 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:53:42.238557    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c\": container with ID starting with b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c not found: ID does not exist" containerID="b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c"
Sep 17 17:53:42.238628 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.238581    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c"} err="failed to get container status \"b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c\": rpc error: code = NotFound desc = could not find container \"b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c\": container with ID starting with b9b24f31418a879d102de186cb22f01ee3dcd69fd4afa6b0d9649b3acc680a8c not found: ID does not exist"
Sep 17 17:53:42.252978 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.252938    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"]
Sep 17 17:53:42.257956 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:42.257937    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfdxj5h"]
Sep 17 17:53:43.181364 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:43.181335    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f724b6-4899-481c-aa7b-a0eb892de18f" path="/var/lib/kubelet/pods/c3f724b6-4899-481c-aa7b-a0eb892de18f/volumes"
Sep 17 17:53:45.299746 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:45.299705    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:48.216165 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.216132    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"]
Sep 17 17:53:48.216547 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.216532    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f724b6-4899-481c-aa7b-a0eb892de18f" containerName="main"
Sep 17 17:53:48.216547 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.216548    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f724b6-4899-481c-aa7b-a0eb892de18f" containerName="main"
Sep 17 17:53:48.216642 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.216623    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f724b6-4899-481c-aa7b-a0eb892de18f" containerName="main"
Sep 17 17:53:48.219712 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.219694    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.222339 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.222314    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 17:53:48.222339 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.222316    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-s7868\""
Sep 17 17:53:48.222628 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.222613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 17:53:48.222805 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.222776    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 17:53:48.222945 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.222817    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:53:48.223444 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.223428    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 17:53:48.233341 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.233315    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"]
Sep 17 17:53:48.238847 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.238827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:48.281801 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.281781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-tmp-dir\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.281902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.281813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-kserve-provision-location\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.281902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.281838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd70069-7279-40de-b34d-85c51363806a-cabundle-cert\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.281902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.281855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd70069-7279-40de-b34d-85c51363806a-tls-certs\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.282052 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.281990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-home\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.282052 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.282019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmtj\" (UniqueName: \"kubernetes.io/projected/7bd70069-7279-40de-b34d-85c51363806a-kube-api-access-mwmtj\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.282137 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.282057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-dshm\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.282137 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.282078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-model-cache\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.382623 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.382601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-home\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.382719 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.382630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmtj\" (UniqueName: \"kubernetes.io/projected/7bd70069-7279-40de-b34d-85c51363806a-kube-api-access-mwmtj\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.382719 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.382652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-dshm\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.382719 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.382672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-model-cache\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.382719 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.382710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-tmp-dir\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.382925 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.382734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-kserve-provision-location\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.382925 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.382767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd70069-7279-40de-b34d-85c51363806a-cabundle-cert\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.382925 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.382790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd70069-7279-40de-b34d-85c51363806a-tls-certs\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.383083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.383062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-home\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.383330 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.383308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-model-cache\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.383412 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.383368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-kserve-provision-location\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.383481 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.383433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-tmp-dir\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.383481 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.383463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd70069-7279-40de-b34d-85c51363806a-cabundle-cert\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.384800 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.384782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-dshm\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.385119 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.385102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd70069-7279-40de-b34d-85c51363806a-tls-certs\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.391906 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.391878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmtj\" (UniqueName: \"kubernetes.io/projected/7bd70069-7279-40de-b34d-85c51363806a-kube-api-access-mwmtj\") pod \"auth-enabled-test-kserve-5d776d77c7-jhrgq\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.530278 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.530224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:53:48.633219 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.633190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:48.813469 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.813445    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"]
Sep 17 17:53:48.814913 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:53:48.814889    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd70069_7279_40de_b34d_85c51363806a.slice/crio-752d87fbd7b6e44a510d2ee91910bad471fbcf3f24d82f29ad038bdce4f4d7d4 WatchSource:0}: Error finding container 752d87fbd7b6e44a510d2ee91910bad471fbcf3f24d82f29ad038bdce4f4d7d4: Status 404 returned error can't find the container with id 752d87fbd7b6e44a510d2ee91910bad471fbcf3f24d82f29ad038bdce4f4d7d4
Sep 17 17:53:48.932297 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:48.932274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:49.275493 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:49.275463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" event={"ID":"7bd70069-7279-40de-b34d-85c51363806a","Type":"ContainerStarted","Data":"752d87fbd7b6e44a510d2ee91910bad471fbcf3f24d82f29ad038bdce4f4d7d4"}
Sep 17 17:53:50.645644 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:50.645613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:52.319267 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:52.319235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:55.314682 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:55.314638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" event={"ID":"7bd70069-7279-40de-b34d-85c51363806a","Type":"ContainerStarted","Data":"ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1"}
Sep 17 17:53:55.872681 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:55.872646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:57.326748 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:57.326716    2581 generic.go:356] "Generic (PLEG): container finished" podID="7bd70069-7279-40de-b34d-85c51363806a" containerID="ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1" exitCode=0
Sep 17 17:53:57.327164 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:57.326789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" event={"ID":"7bd70069-7279-40de-b34d-85c51363806a","Type":"ContainerDied","Data":"ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1"}
Sep 17 17:53:57.459009 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:57.458961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:57.559533 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:57.559507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:57.765857 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:57.765826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:59.823179 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:59.823150    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:53:59.823819 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:53:59.823799    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:54:07.348132 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:07.348056    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:07.415047 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:07.415008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:24.499100 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:24.499068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" event={"ID":"7bd70069-7279-40de-b34d-85c51363806a","Type":"ContainerStarted","Data":"66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1"}
Sep 17 17:54:24.525855 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:24.525794    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podStartSLOduration=2.043409593 podStartE2EDuration="36.525774633s" totalImagesPullingTime="32.430026907s" totalInitContainerRuntime="2.052338133s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:24.521310195 +0000 UTC m=+1227.445782152" watchObservedRunningTime="2026-09-17 17:54:24.525774633 +0000 UTC m=+1227.450246590"
Sep 17 17:54:24.531395 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:24.531363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:54:28.530721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:28.530693    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:54:28.532169 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:28.532139    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:54:38.531145 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:38.531104    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:54:38.765985 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:38.765937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:38.840521 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:38.840445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:38.908803 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:38.908774    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:48.176339 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:48.176261    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:48.530679 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:48.530632    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:54:58.531261 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:54:58.531221    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:55:08.530595 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:08.530554    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:55:18.531270 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:18.531223    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:55:28.531369 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:28.531319    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:55:28.778045 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:28.778016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:28.875214 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:28.875147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:31.695849 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:31.695813    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:31.969387 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:31.969351    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:35.639391 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:35.639361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:38.531132 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:38.531086    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:55:41.643328 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:41.643291    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:41.741929 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:41.741892    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:45.111363 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:45.111323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:48.531431 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:48.531381    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:55:58.531345 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:55:58.531295    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:56:06.897270 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:06.897224    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:07.899561 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:07.899524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:08.531108 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:08.531069    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:56:18.540872 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:18.540781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:56:18.548681 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:18.548654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:56:24.215984 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:24.215922    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:24.220860 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:24.220833    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"]
Sep 17 17:56:24.313080 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:24.313030    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:39.242615 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:39.242571    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" containerID="cri-o://66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1" gracePeriod=45
Sep 17 17:56:39.542265 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:39.542182    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:56:39.978393 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:39.978372    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:56:40.098442 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.098419    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-tmp-dir\") pod \"7bd70069-7279-40de-b34d-85c51363806a\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") "
Sep 17 17:56:40.098584 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.098456    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-home\") pod \"7bd70069-7279-40de-b34d-85c51363806a\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") "
Sep 17 17:56:40.098584 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.098479    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-dshm\") pod \"7bd70069-7279-40de-b34d-85c51363806a\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") "
Sep 17 17:56:40.098584 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.098503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd70069-7279-40de-b34d-85c51363806a-cabundle-cert\") pod \"7bd70069-7279-40de-b34d-85c51363806a\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") "
Sep 17 17:56:40.098584 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.098541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd70069-7279-40de-b34d-85c51363806a-tls-certs\") pod \"7bd70069-7279-40de-b34d-85c51363806a\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") "
Sep 17 17:56:40.098584 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.098571    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwmtj\" (UniqueName: \"kubernetes.io/projected/7bd70069-7279-40de-b34d-85c51363806a-kube-api-access-mwmtj\") pod \"7bd70069-7279-40de-b34d-85c51363806a\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") "
Sep 17 17:56:40.098849 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.098593    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-model-cache\") pod \"7bd70069-7279-40de-b34d-85c51363806a\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") "
Sep 17 17:56:40.098849 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.098613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-kserve-provision-location\") pod \"7bd70069-7279-40de-b34d-85c51363806a\" (UID: \"7bd70069-7279-40de-b34d-85c51363806a\") "
Sep 17 17:56:40.099145 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.098924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd70069-7279-40de-b34d-85c51363806a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bd70069-7279-40de-b34d-85c51363806a" (UID: "7bd70069-7279-40de-b34d-85c51363806a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:40.099235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.099206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bd70069-7279-40de-b34d-85c51363806a" (UID: "7bd70069-7279-40de-b34d-85c51363806a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:40.099235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.099222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-home" (OuterVolumeSpecName: "home") pod "7bd70069-7279-40de-b34d-85c51363806a" (UID: "7bd70069-7279-40de-b34d-85c51363806a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:40.100753 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.100729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd70069-7279-40de-b34d-85c51363806a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bd70069-7279-40de-b34d-85c51363806a" (UID: "7bd70069-7279-40de-b34d-85c51363806a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:40.101014 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.100977    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-dshm" (OuterVolumeSpecName: "dshm") pod "7bd70069-7279-40de-b34d-85c51363806a" (UID: "7bd70069-7279-40de-b34d-85c51363806a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:40.101132 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.101070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd70069-7279-40de-b34d-85c51363806a-kube-api-access-mwmtj" (OuterVolumeSpecName: "kube-api-access-mwmtj") pod "7bd70069-7279-40de-b34d-85c51363806a" (UID: "7bd70069-7279-40de-b34d-85c51363806a"). InnerVolumeSpecName "kube-api-access-mwmtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:40.110213 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.110189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bd70069-7279-40de-b34d-85c51363806a" (UID: "7bd70069-7279-40de-b34d-85c51363806a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:40.117916 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.117896    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bd70069-7279-40de-b34d-85c51363806a" (UID: "7bd70069-7279-40de-b34d-85c51363806a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:40.199842 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.199822    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.199842 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.199841    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.199986 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.199850    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.199986 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.199857    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd70069-7279-40de-b34d-85c51363806a-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.199986 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.199864    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd70069-7279-40de-b34d-85c51363806a-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.199986 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.199872    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmtj\" (UniqueName: \"kubernetes.io/projected/7bd70069-7279-40de-b34d-85c51363806a-kube-api-access-mwmtj\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.199986 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.199880    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.199986 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.199887    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd70069-7279-40de-b34d-85c51363806a-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.253864 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.253842    2581 generic.go:356] "Generic (PLEG): container finished" podID="7bd70069-7279-40de-b34d-85c51363806a" containerID="66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1" exitCode=0
Sep 17 17:56:40.254167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.253908    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"
Sep 17 17:56:40.254167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.253929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" event={"ID":"7bd70069-7279-40de-b34d-85c51363806a","Type":"ContainerDied","Data":"66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1"}
Sep 17 17:56:40.254167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.253997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq" event={"ID":"7bd70069-7279-40de-b34d-85c51363806a","Type":"ContainerDied","Data":"752d87fbd7b6e44a510d2ee91910bad471fbcf3f24d82f29ad038bdce4f4d7d4"}
Sep 17 17:56:40.254167 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.254022    2581 scope.go:117] "RemoveContainer" containerID="66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1"
Sep 17 17:56:40.263168 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.263151    2581 scope.go:117] "RemoveContainer" containerID="ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1"
Sep 17 17:56:40.271083 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.271066    2581 scope.go:117] "RemoveContainer" containerID="66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1"
Sep 17 17:56:40.271351 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:56:40.271334    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1\": container with ID starting with 66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1 not found: ID does not exist" containerID="66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1"
Sep 17 17:56:40.271407 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.271359    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1"} err="failed to get container status \"66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1\": rpc error: code = NotFound desc = could not find container \"66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1\": container with ID starting with 66d86d77a6beba09875fdf94ed61e8952b81af546c992b671754cb14aef864d1 not found: ID does not exist"
Sep 17 17:56:40.271407 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.271376    2581 scope.go:117] "RemoveContainer" containerID="ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1"
Sep 17 17:56:40.271622 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:56:40.271604    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1\": container with ID starting with ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1 not found: ID does not exist" containerID="ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1"
Sep 17 17:56:40.271674 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.271632    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1"} err="failed to get container status \"ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1\": rpc error: code = NotFound desc = could not find container \"ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1\": container with ID starting with ba92d49627fa6ca97771a11388c1684adedc2f857cba55d6604759755c7d50d1 not found: ID does not exist"
Sep 17 17:56:40.279746 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.279727    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"]
Sep 17 17:56:40.284050 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:40.284030    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5d776d77c7-jhrgq"]
Sep 17 17:56:41.181251 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:41.181217    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd70069-7279-40de-b34d-85c51363806a" path="/var/lib/kubelet/pods/7bd70069-7279-40de-b34d-85c51363806a/volumes"
Sep 17 17:56:56.998499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:56.998459    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:57.101260 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:56:57.101232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:01.498073 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:01.498033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:01.554122 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:01.554063    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:03.606021 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:03.605987    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:09.824470 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:09.824440    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:09.901117 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:09.901080    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:14.180298 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:14.180260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:14.281061 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:14.281027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:19.765596 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.765562    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"]
Sep 17 17:57:19.765985 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.765935    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="storage-initializer"
Sep 17 17:57:19.765985 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.765947    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="storage-initializer"
Sep 17 17:57:19.765985 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.765959    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main"
Sep 17 17:57:19.766098 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.765990    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main"
Sep 17 17:57:19.766098 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.766060    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd70069-7279-40de-b34d-85c51363806a" containerName="main"
Sep 17 17:57:19.769075 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.769060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.771705 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.771680    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 17:57:19.771842 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.771794    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 17:57:19.771842 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.771798    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:57:19.772960 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.772939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 17:57:19.772960 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.772950    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 17:57:19.773164 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.773058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-86qsh\""
Sep 17 17:57:19.773235 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.773220    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:19.784679 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.784657    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"]
Sep 17 17:57:19.898214 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.898193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.898315 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.898230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-model-cache\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.898315 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.898249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.898390 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.898315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-home\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.898390 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.898346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.898390 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.898368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.898390 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.898385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pss8\" (UniqueName: \"kubernetes.io/projected/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kube-api-access-6pss8\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.898525 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.898455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-dshm\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.999806 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.999778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.999927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.999837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-model-cache\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.999927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.999864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:19.999927 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:19.999899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-home\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.000079 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.000008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.000079 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.000049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.000176 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.000076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pss8\" (UniqueName: \"kubernetes.io/projected/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kube-api-access-6pss8\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.000312 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.000284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-model-cache\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.000463 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.000438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-dshm\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.000596 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.000541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-home\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.000596 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.000553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.000848 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.000828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.000902 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.000872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.002602 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.002578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.002717 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.002670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-dshm\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.017413 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.017359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pss8\" (UniqueName: \"kubernetes.io/projected/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kube-api-access-6pss8\") pod \"tokenizer-clean-path-test-kserve-7948d59869-4vnkz\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.047960 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.047933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:20.080726 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.080707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:20.150155 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.150125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:20.356332 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.356260    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"]
Sep 17 17:57:20.359528 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:57:20.359500    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4a7bff_edf8_4fc9_902c_fc31956b10c2.slice/crio-04c1b7b63606536ab41f2ae13a8b8171c0b3d74b1b206ca979af03480c92c6cc WatchSource:0}: Error finding container 04c1b7b63606536ab41f2ae13a8b8171c0b3d74b1b206ca979af03480c92c6cc: Status 404 returned error can't find the container with id 04c1b7b63606536ab41f2ae13a8b8171c0b3d74b1b206ca979af03480c92c6cc
Sep 17 17:57:20.465622 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.465537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" event={"ID":"7d4a7bff-edf8-4fc9-902c-fc31956b10c2","Type":"ContainerStarted","Data":"9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f"}
Sep 17 17:57:20.465622 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:20.465601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" event={"ID":"7d4a7bff-edf8-4fc9-902c-fc31956b10c2","Type":"ContainerStarted","Data":"04c1b7b63606536ab41f2ae13a8b8171c0b3d74b1b206ca979af03480c92c6cc"}
Sep 17 17:57:21.896400 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:21.896358    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:23.483072 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:23.483037    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerID="9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f" exitCode=0
Sep 17 17:57:23.483465 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:23.483089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" event={"ID":"7d4a7bff-edf8-4fc9-902c-fc31956b10c2","Type":"ContainerDied","Data":"9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f"}
Sep 17 17:57:24.490748 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:24.490713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" event={"ID":"7d4a7bff-edf8-4fc9-902c-fc31956b10c2","Type":"ContainerStarted","Data":"9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46"}
Sep 17 17:57:24.512857 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:24.512810    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" podStartSLOduration=3.011515712 podStartE2EDuration="5.512795814s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.501280102s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:24.511523741 +0000 UTC m=+1407.435995700" watchObservedRunningTime="2026-09-17 17:57:24.512795814 +0000 UTC m=+1407.437267771"
Sep 17 17:57:25.081383 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:25.081355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:27.392033 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:27.391984    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:27.843836 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:27.843806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:27.944551 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:27.944522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:28.183949 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:28.183879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:28.244516 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:28.244490    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:30.082034 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:30.081998    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:30.094560 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:30.094531    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:30.534152 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:30.534125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:57:31.285166 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:31.285136    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:38.119298 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.119263    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"]
Sep 17 17:57:38.122139 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.122120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.124777 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.124745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:57:38.124948 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.124744    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 17:57:38.124948 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.124813    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 17:57:38.125112 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.124953    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 17:57:38.125162 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.125114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 17:57:38.125162 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.125121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-7w2kn\""
Sep 17 17:57:38.133032 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.133010    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"]
Sep 17 17:57:38.149223 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.149194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tls-certs\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.149332 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.149226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tmp-dir\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.149332 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.149249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.149421 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.149364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-dshm\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.149421 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.149408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6tqx\" (UniqueName: \"kubernetes.io/projected/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kube-api-access-n6tqx\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.149503 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.149473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-model-cache\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.149503 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.149499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-home\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.149603 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.149588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38285e69-8eb5-4da5-9e4c-f7d52fb55213-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.207542 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.207507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:38.250721 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.250695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tls-certs\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.250821 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.250727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tmp-dir\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.250821 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.250745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.250821 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.250769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-dshm\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.250821 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.250790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6tqx\" (UniqueName: \"kubernetes.io/projected/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kube-api-access-n6tqx\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.251044 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.250827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-model-cache\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.251044 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.250861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-home\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.251044 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.250935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38285e69-8eb5-4da5-9e4c-f7d52fb55213-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.251188 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.251149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tmp-dir\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.251188 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.251173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.251312 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.251292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-home\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.251382 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.251360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-model-cache\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.251683 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.251665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38285e69-8eb5-4da5-9e4c-f7d52fb55213-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.253499 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.253483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-dshm\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.253753 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.253737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tls-certs\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.261476 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.261446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6tqx\" (UniqueName: \"kubernetes.io/projected/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kube-api-access-n6tqx\") pod \"auth-invalid-token-test-kserve-8486457b5c-f4br8\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.415210 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.415137    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:38.433325 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.433295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:38.710917 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:38.710894    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"]
Sep 17 17:57:38.712387 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:57:38.712362    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38285e69_8eb5_4da5_9e4c_f7d52fb55213.slice/crio-7970ce7d68b8ebbd6c8ce43c732e59439f065b372e9f9b7cbcda83bc649537db WatchSource:0}: Error finding container 7970ce7d68b8ebbd6c8ce43c732e59439f065b372e9f9b7cbcda83bc649537db: Status 404 returned error can't find the container with id 7970ce7d68b8ebbd6c8ce43c732e59439f065b372e9f9b7cbcda83bc649537db
Sep 17 17:57:39.571640 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:39.571595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" event={"ID":"38285e69-8eb5-4da5-9e4c-f7d52fb55213","Type":"ContainerStarted","Data":"acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4"}
Sep 17 17:57:39.572017 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:39.571667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" event={"ID":"38285e69-8eb5-4da5-9e4c-f7d52fb55213","Type":"ContainerStarted","Data":"7970ce7d68b8ebbd6c8ce43c732e59439f065b372e9f9b7cbcda83bc649537db"}
Sep 17 17:57:41.583057 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:41.583028    2581 generic.go:356] "Generic (PLEG): container finished" podID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerID="acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4" exitCode=0
Sep 17 17:57:41.583411 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:41.583103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" event={"ID":"38285e69-8eb5-4da5-9e4c-f7d52fb55213","Type":"ContainerDied","Data":"acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4"}
Sep 17 17:57:42.590361 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:42.590327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" event={"ID":"38285e69-8eb5-4da5-9e4c-f7d52fb55213","Type":"ContainerStarted","Data":"ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90"}
Sep 17 17:57:42.614193 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:42.614143    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podStartSLOduration=2.474156758 podStartE2EDuration="4.614127741s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.139970983s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:42.612648707 +0000 UTC m=+1425.537120664" watchObservedRunningTime="2026-09-17 17:57:42.614127741 +0000 UTC m=+1425.538599698"
Sep 17 17:57:43.434356 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:43.434326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:48.433823 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:48.433788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:57:48.435290 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:48.435262    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:57:52.654248 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:52.654214    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:58.433745 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:57:58.433695    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:58:08.433842 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:08.433800    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:58:18.434569 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:18.434527    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:58:28.433819 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:28.433774    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:58:38.434175 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:38.434127    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:58:41.950500 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:41.950463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:41.964256 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:41.964231    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"]
Sep 17 17:58:41.974120 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:58:41.974085    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:58:41.974120 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:41.974120 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:41.974120 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46" cmd=["/bin/sleep","15"]
Sep 17 17:58:41.974438 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:41.974144    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" message="" err=<
Sep 17 17:58:41.974438 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:41.974438 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:41.974438 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 17 17:58:41.974438 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:58:41.974162    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:58:41.974438 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:58:41.974438 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:58:41.974438 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" podUID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerName="main" containerID="cri-o://9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46"
Sep 17 17:58:41.974438 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:41.974196    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" podUID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerName="main" containerID="cri-o://9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46" gracePeriod=60
Sep 17 17:58:42.046825 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.046797    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:42.523727 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.523683    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" podUID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 17:58:42.727028 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.727004    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:58:42.795651 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-model-cache\") pod \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") "
Sep 17 17:58:42.795775 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795656    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tmp-dir\") pod \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") "
Sep 17 17:58:42.795775 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795677    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-cabundle-cert\") pod \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") "
Sep 17 17:58:42.795775 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795730    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pss8\" (UniqueName: \"kubernetes.io/projected/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kube-api-access-6pss8\") pod \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") "
Sep 17 17:58:42.795775 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tls-certs\") pod \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") "
Sep 17 17:58:42.796000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795804    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kserve-provision-location\") pod \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") "
Sep 17 17:58:42.796000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795810    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d4a7bff-edf8-4fc9-902c-fc31956b10c2" (UID: "7d4a7bff-edf8-4fc9-902c-fc31956b10c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.796000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795854    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-home\") pod \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") "
Sep 17 17:58:42.796000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-dshm\") pod \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\" (UID: \"7d4a7bff-edf8-4fc9-902c-fc31956b10c2\") "
Sep 17 17:58:42.796000 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795906    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d4a7bff-edf8-4fc9-902c-fc31956b10c2" (UID: "7d4a7bff-edf8-4fc9-902c-fc31956b10c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.796267 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.795995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d4a7bff-edf8-4fc9-902c-fc31956b10c2" (UID: "7d4a7bff-edf8-4fc9-902c-fc31956b10c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:42.796267 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.796146    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-home" (OuterVolumeSpecName: "home") pod "7d4a7bff-edf8-4fc9-902c-fc31956b10c2" (UID: "7d4a7bff-edf8-4fc9-902c-fc31956b10c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.796267 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.796214    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.796267 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.796237    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.796267 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.796249    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.798528 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.798503    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-dshm" (OuterVolumeSpecName: "dshm") pod "7d4a7bff-edf8-4fc9-902c-fc31956b10c2" (UID: "7d4a7bff-edf8-4fc9-902c-fc31956b10c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.798528 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.798509    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kube-api-access-6pss8" (OuterVolumeSpecName: "kube-api-access-6pss8") pod "7d4a7bff-edf8-4fc9-902c-fc31956b10c2" (UID: "7d4a7bff-edf8-4fc9-902c-fc31956b10c2"). InnerVolumeSpecName "kube-api-access-6pss8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:42.798698 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.798503    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d4a7bff-edf8-4fc9-902c-fc31956b10c2" (UID: "7d4a7bff-edf8-4fc9-902c-fc31956b10c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:42.817335 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.817308    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d4a7bff-edf8-4fc9-902c-fc31956b10c2" (UID: "7d4a7bff-edf8-4fc9-902c-fc31956b10c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:42.897272 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.897243    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pss8\" (UniqueName: \"kubernetes.io/projected/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kube-api-access-6pss8\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.897272 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.897267    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.897272 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.897276    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.897480 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.897285    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.897480 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.897293    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4a7bff-edf8-4fc9-902c-fc31956b10c2-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:58:42.932168 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.932139    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerID="9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46" exitCode=0
Sep 17 17:58:42.932306 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.932226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" event={"ID":"7d4a7bff-edf8-4fc9-902c-fc31956b10c2","Type":"ContainerDied","Data":"9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46"}
Sep 17 17:58:42.932306 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.932286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz" event={"ID":"7d4a7bff-edf8-4fc9-902c-fc31956b10c2","Type":"ContainerDied","Data":"04c1b7b63606536ab41f2ae13a8b8171c0b3d74b1b206ca979af03480c92c6cc"}
Sep 17 17:58:42.932379 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.932308    2581 scope.go:117] "RemoveContainer" containerID="9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46"
Sep 17 17:58:42.932379 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.932237    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"
Sep 17 17:58:42.942857 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.942841    2581 scope.go:117] "RemoveContainer" containerID="9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f"
Sep 17 17:58:42.950484 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.950468    2581 scope.go:117] "RemoveContainer" containerID="9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46"
Sep 17 17:58:42.950748 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:58:42.950732    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46\": container with ID starting with 9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46 not found: ID does not exist" containerID="9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46"
Sep 17 17:58:42.951005 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.950756    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46"} err="failed to get container status \"9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46\": rpc error: code = NotFound desc = could not find container \"9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46\": container with ID starting with 9a778b78ccfcd237ded58e6f4f01ea0428c33476d275f90760d10dcb95c96d46 not found: ID does not exist"
Sep 17 17:58:42.951005 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.950769    2581 scope.go:117] "RemoveContainer" containerID="9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f"
Sep 17 17:58:42.951072 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:58:42.951025    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f\": container with ID starting with 9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f not found: ID does not exist" containerID="9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f"
Sep 17 17:58:42.951072 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.951044    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f"} err="failed to get container status \"9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f\": rpc error: code = NotFound desc = could not find container \"9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f\": container with ID starting with 9636bbb29632f50d3d7cafaeecd9260a67d2e9ff270061ef8b24e6a91d1d384f not found: ID does not exist"
Sep 17 17:58:42.963439 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.963419    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"]
Sep 17 17:58:42.970709 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:42.970690    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7948d59869-4vnkz"]
Sep 17 17:58:43.181586 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:43.181528    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" path="/var/lib/kubelet/pods/7d4a7bff-edf8-4fc9-902c-fc31956b10c2/volumes"
Sep 17 17:58:48.434310 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:48.434264    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:58:58.434195 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:58.434148    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:58:59.868568 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:59.868543    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:58:59.873192 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:58:59.873171    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 17:59:08.434620 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:08.434582    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:59:18.434102 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:18.434018    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 17:59:28.444090 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:28.444053    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:59:28.452631 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:28.452600    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:59:30.514224 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:30.514191    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"]
Sep 17 17:59:31.688404 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:31.688360    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:31.793202 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:31.793169    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:32.078177 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:32.078148    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:32.175121 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:32.175093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:38.991799 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:38.991761    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:44.829590 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:44.829555    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:44.930227 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:44.930186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:45.535548 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:45.535492    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main" containerID="cri-o://ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90" gracePeriod=45
Sep 17 17:59:46.206301 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.206279    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:59:46.276048 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.276025    2581 generic.go:356] "Generic (PLEG): container finished" podID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerID="ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90" exitCode=0
Sep 17 17:59:46.276177 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.276073    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"
Sep 17 17:59:46.276177 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.276115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" event={"ID":"38285e69-8eb5-4da5-9e4c-f7d52fb55213","Type":"ContainerDied","Data":"ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90"}
Sep 17 17:59:46.276298 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.276199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8" event={"ID":"38285e69-8eb5-4da5-9e4c-f7d52fb55213","Type":"ContainerDied","Data":"7970ce7d68b8ebbd6c8ce43c732e59439f065b372e9f9b7cbcda83bc649537db"}
Sep 17 17:59:46.276298 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.276233    2581 scope.go:117] "RemoveContainer" containerID="ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90"
Sep 17 17:59:46.286493 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.286475    2581 scope.go:117] "RemoveContainer" containerID="acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4"
Sep 17 17:59:46.294380 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.294366    2581 scope.go:117] "RemoveContainer" containerID="ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90"
Sep 17 17:59:46.294624 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:59:46.294601    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90\": container with ID starting with ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90 not found: ID does not exist" containerID="ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90"
Sep 17 17:59:46.294681 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.294635    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90"} err="failed to get container status \"ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90\": rpc error: code = NotFound desc = could not find container \"ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90\": container with ID starting with ebcb61898ce47e1e13eec88faa5d0bd991a1525597f152ddaacc9b9f001f5c90 not found: ID does not exist"
Sep 17 17:59:46.294681 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.294650    2581 scope.go:117] "RemoveContainer" containerID="acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4"
Sep 17 17:59:46.294885 ip-10-0-133-194 kubenswrapper[2581]: E0917 17:59:46.294868    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4\": container with ID starting with acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4 not found: ID does not exist" containerID="acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4"
Sep 17 17:59:46.294925 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.294892    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4"} err="failed to get container status \"acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4\": rpc error: code = NotFound desc = could not find container \"acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4\": container with ID starting with acefc436538e451a045355bfc9f30138ebf85c08ec135f46998f0a755f85adc4 not found: ID does not exist"
Sep 17 17:59:46.327197 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327175    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38285e69-8eb5-4da5-9e4c-f7d52fb55213-cabundle-cert\") pod \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") "
Sep 17 17:59:46.327269 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327208    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kserve-provision-location\") pod \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") "
Sep 17 17:59:46.327269 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327261    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tls-certs\") pod \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") "
Sep 17 17:59:46.327452 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327432    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-dshm\") pod \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") "
Sep 17 17:59:46.327487 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327467    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6tqx\" (UniqueName: \"kubernetes.io/projected/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kube-api-access-n6tqx\") pod \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") "
Sep 17 17:59:46.327527 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327486    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-home\") pod \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") "
Sep 17 17:59:46.327527 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327498    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38285e69-8eb5-4da5-9e4c-f7d52fb55213-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38285e69-8eb5-4da5-9e4c-f7d52fb55213" (UID: "38285e69-8eb5-4da5-9e4c-f7d52fb55213"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:46.327612 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-model-cache\") pod \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") "
Sep 17 17:59:46.327612 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tmp-dir\") pod \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\" (UID: \"38285e69-8eb5-4da5-9e4c-f7d52fb55213\") "
Sep 17 17:59:46.328318 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.327804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-model-cache" (OuterVolumeSpecName: "model-cache") pod "38285e69-8eb5-4da5-9e4c-f7d52fb55213" (UID: "38285e69-8eb5-4da5-9e4c-f7d52fb55213"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:46.328318 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.328039    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:59:46.328318 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.328058    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38285e69-8eb5-4da5-9e4c-f7d52fb55213-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:59:46.328318 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.328262    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-home" (OuterVolumeSpecName: "home") pod "38285e69-8eb5-4da5-9e4c-f7d52fb55213" (UID: "38285e69-8eb5-4da5-9e4c-f7d52fb55213"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:46.329684 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.329661    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-dshm" (OuterVolumeSpecName: "dshm") pod "38285e69-8eb5-4da5-9e4c-f7d52fb55213" (UID: "38285e69-8eb5-4da5-9e4c-f7d52fb55213"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:46.329863 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.329838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kube-api-access-n6tqx" (OuterVolumeSpecName: "kube-api-access-n6tqx") pod "38285e69-8eb5-4da5-9e4c-f7d52fb55213" (UID: "38285e69-8eb5-4da5-9e4c-f7d52fb55213"). InnerVolumeSpecName "kube-api-access-n6tqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:46.329863 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.329846    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38285e69-8eb5-4da5-9e4c-f7d52fb55213" (UID: "38285e69-8eb5-4da5-9e4c-f7d52fb55213"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:46.345917 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.345873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38285e69-8eb5-4da5-9e4c-f7d52fb55213" (UID: "38285e69-8eb5-4da5-9e4c-f7d52fb55213"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:46.355400 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.355375    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38285e69-8eb5-4da5-9e4c-f7d52fb55213" (UID: "38285e69-8eb5-4da5-9e4c-f7d52fb55213"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:46.428677 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.428653    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:59:46.428677 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.428676    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:59:46.428794 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.428686    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6tqx\" (UniqueName: \"kubernetes.io/projected/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kube-api-access-n6tqx\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:59:46.428794 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.428696    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:59:46.428794 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.428705    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:59:46.428794 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.428714    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38285e69-8eb5-4da5-9e4c-f7d52fb55213-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 17:59:46.602774 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.602751    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"]
Sep 17 17:59:46.618060 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:46.618040    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8486457b5c-f4br8"]
Sep 17 17:59:47.181669 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:47.181643    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" path="/var/lib/kubelet/pods/38285e69-8eb5-4da5-9e4c-f7d52fb55213/volumes"
Sep 17 17:59:57.437600 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.437564    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"]
Sep 17 17:59:57.438109 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438091    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="storage-initializer"
Sep 17 17:59:57.438183 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438112    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="storage-initializer"
Sep 17 17:59:57.438183 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438132    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerName="storage-initializer"
Sep 17 17:59:57.438183 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438140    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerName="storage-initializer"
Sep 17 17:59:57.438183 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438151    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerName="main"
Sep 17 17:59:57.438183 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438159    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerName="main"
Sep 17 17:59:57.438500 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438191    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main"
Sep 17 17:59:57.438500 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438199    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main"
Sep 17 17:59:57.438500 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438298    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38285e69-8eb5-4da5-9e4c-f7d52fb55213" containerName="main"
Sep 17 17:59:57.438500 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.438311    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d4a7bff-edf8-4fc9-902c-fc31956b10c2" containerName="main"
Sep 17 17:59:57.441497 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.441477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.444077 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.444049    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 17:59:57.444077 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.444059    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 17:59:57.444213 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.444104    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 17:59:57.445296 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.445278    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-qbq5r\""
Sep 17 17:59:57.445366 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.445286    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:59:57.445366 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.445332    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 17:59:57.451046 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.451020    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"]
Sep 17 17:59:57.457815 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.457795    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:57.616676 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.616655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.616789 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.616687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6svp7\" (UniqueName: \"kubernetes.io/projected/19cb1349-2702-4bd5-af3c-0af3ced5f261-kube-api-access-6svp7\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.616789 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.616705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-home\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.616789 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.616725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19cb1349-2702-4bd5-af3c-0af3ced5f261-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.616917 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.616794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-dshm\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.616917 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.616820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-model-cache\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.616917 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.616889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cb1349-2702-4bd5-af3c-0af3ced5f261-tls-certs\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.616917 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.616906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.715047 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.715025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:57.718183 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-dshm\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718300 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-model-cache\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718300 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cb1349-2702-4bd5-af3c-0af3ced5f261-tls-certs\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718300 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718462 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718462 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6svp7\" (UniqueName: \"kubernetes.io/projected/19cb1349-2702-4bd5-af3c-0af3ced5f261-kube-api-access-6svp7\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718462 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-home\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718622 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19cb1349-2702-4bd5-af3c-0af3ced5f261-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718682 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-model-cache\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718748 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.718824 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.718807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-home\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.719299 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.719277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19cb1349-2702-4bd5-af3c-0af3ced5f261-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.719481 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.719461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.720501 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.720482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-dshm\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.720765 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.720744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cb1349-2702-4bd5-af3c-0af3ced5f261-tls-certs\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.730311 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.730288    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6svp7\" (UniqueName: \"kubernetes.io/projected/19cb1349-2702-4bd5-af3c-0af3ced5f261-kube-api-access-6svp7\") pod \"tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.753008 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.752984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 17:59:57.811071 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:57.811044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:58.029959 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:58.029886    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"]
Sep 17 17:59:58.032911 ip-10-0-133-194 kubenswrapper[2581]: W0917 17:59:58.032888    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cb1349_2702_4bd5_af3c_0af3ced5f261.slice/crio-f6026929b526bc44d7abb3de4f3d03d834668e08d230dd2f0f06329a40bba426 WatchSource:0}: Error finding container f6026929b526bc44d7abb3de4f3d03d834668e08d230dd2f0f06329a40bba426: Status 404 returned error can't find the container with id f6026929b526bc44d7abb3de4f3d03d834668e08d230dd2f0f06329a40bba426
Sep 17 17:59:58.344728 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:58.344629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" event={"ID":"19cb1349-2702-4bd5-af3c-0af3ced5f261","Type":"ContainerStarted","Data":"e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976"}
Sep 17 17:59:58.344888 ip-10-0-133-194 kubenswrapper[2581]: I0917 17:59:58.344739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" event={"ID":"19cb1349-2702-4bd5-af3c-0af3ced5f261","Type":"ContainerStarted","Data":"f6026929b526bc44d7abb3de4f3d03d834668e08d230dd2f0f06329a40bba426"}
Sep 17 18:00:00.356871 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:00.356842    2581 generic.go:356] "Generic (PLEG): container finished" podID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerID="e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976" exitCode=0
Sep 17 18:00:00.357194 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:00.356906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" event={"ID":"19cb1349-2702-4bd5-af3c-0af3ced5f261","Type":"ContainerDied","Data":"e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976"}
Sep 17 18:00:01.363322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:01.363290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" event={"ID":"19cb1349-2702-4bd5-af3c-0af3ced5f261","Type":"ContainerStarted","Data":"d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe"}
Sep 17 18:00:01.388534 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:01.388486    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" podStartSLOduration=2.248386266 podStartE2EDuration="4.38847144s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.140085174s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:01.385996691 +0000 UTC m=+1564.310468649" watchObservedRunningTime="2026-09-17 18:00:01.38847144 +0000 UTC m=+1564.312943398"
Sep 17 18:00:01.753300 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:01.753260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 18:00:07.753674 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:07.753627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 18:00:07.765882 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:07.765853    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 18:00:08.273956 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:08.273919    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:08.412292 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:08.412266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 18:00:20.176856 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:20.176825    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:20.269116 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:20.269089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:27.772292 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:27.772256    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:32.637824 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.637786    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"]
Sep 17 18:00:32.641269 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.641246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.646732 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.646707    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-b95jf\""
Sep 17 18:00:32.648082 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.648058    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 18:00:32.648731 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.648703    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:00:32.648846 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.648712    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 18:00:32.649453 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.649430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 18:00:32.649564 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.649451    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 18:00:32.673196 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.673175    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"]
Sep 17 18:00:32.692302 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.692279    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:32.701144 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.701116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-model-cache\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.701232 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.701150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2kxm\" (UniqueName: \"kubernetes.io/projected/7f3004b7-a124-481c-90d9-423237d06156-kube-api-access-v2kxm\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.701295 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.701245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3004b7-a124-481c-90d9-423237d06156-tls-certs\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.701334 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.701283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-tmp-dir\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.701375 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.701338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-home\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.701375 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.701353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3004b7-a124-481c-90d9-423237d06156-cabundle-cert\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.701449 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.701384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-kserve-provision-location\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.701449 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.701422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-dshm\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802367 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-model-cache\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802459 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2kxm\" (UniqueName: \"kubernetes.io/projected/7f3004b7-a124-481c-90d9-423237d06156-kube-api-access-v2kxm\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802459 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3004b7-a124-481c-90d9-423237d06156-tls-certs\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802459 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-tmp-dir\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802459 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-home\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802601 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3004b7-a124-481c-90d9-423237d06156-cabundle-cert\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802657 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-kserve-provision-location\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802657 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-dshm\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802806 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-model-cache\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802881 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-home\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.802881 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-tmp-dir\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.803014 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.802936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-kserve-provision-location\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.803309 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.803285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3004b7-a124-481c-90d9-423237d06156-cabundle-cert\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.804887 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.804864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-dshm\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.804960 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.804916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3004b7-a124-481c-90d9-423237d06156-tls-certs\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.819425 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.819401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2kxm\" (UniqueName: \"kubernetes.io/projected/7f3004b7-a124-481c-90d9-423237d06156-kube-api-access-v2kxm\") pod \"auth-disabled-test-kserve-849ff9f6d6-lvst7\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:32.950756 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.950729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:32.952688 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:32.952671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:33.189301 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:33.189269    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:33.240325 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:33.240301    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"]
Sep 17 18:00:33.241327 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:00:33.241305    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3004b7_a124_481c_90d9_423237d06156.slice/crio-2afaa36c4ba052247e9ab61bf59361a426213b50cd70ec1517c2d8fe84197ce3 WatchSource:0}: Error finding container 2afaa36c4ba052247e9ab61bf59361a426213b50cd70ec1517c2d8fe84197ce3: Status 404 returned error can't find the container with id 2afaa36c4ba052247e9ab61bf59361a426213b50cd70ec1517c2d8fe84197ce3
Sep 17 18:00:33.294100 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:33.294075    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:33.538167 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:33.538080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" event={"ID":"7f3004b7-a124-481c-90d9-423237d06156","Type":"ContainerStarted","Data":"7a61b31c74615e94687cc30faf434644e72273e80d6493c4ed6f38d9b59b7621"}
Sep 17 18:00:33.538167 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:33.538146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" event={"ID":"7f3004b7-a124-481c-90d9-423237d06156","Type":"ContainerStarted","Data":"2afaa36c4ba052247e9ab61bf59361a426213b50cd70ec1517c2d8fe84197ce3"}
Sep 17 18:00:35.550066 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:35.550040    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f3004b7-a124-481c-90d9-423237d06156" containerID="7a61b31c74615e94687cc30faf434644e72273e80d6493c4ed6f38d9b59b7621" exitCode=0
Sep 17 18:00:35.550392 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:35.550117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" event={"ID":"7f3004b7-a124-481c-90d9-423237d06156","Type":"ContainerDied","Data":"7a61b31c74615e94687cc30faf434644e72273e80d6493c4ed6f38d9b59b7621"}
Sep 17 18:00:36.558266 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:36.558233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" event={"ID":"7f3004b7-a124-481c-90d9-423237d06156","Type":"ContainerStarted","Data":"6343e6d68972e5ef3566488fd29270821ebe9268e516e0f4e3a6652718a7a566"}
Sep 17 18:00:36.582393 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:36.582351    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podStartSLOduration=2.405183096 podStartE2EDuration="4.582337149s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.177154053s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:36.581309954 +0000 UTC m=+1599.505781910" watchObservedRunningTime="2026-09-17 18:00:36.582337149 +0000 UTC m=+1599.506809106"
Sep 17 18:00:36.953062 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:36.953029    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:42.953545 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:42.953435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:00:42.954953 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:42.954913    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:00:43.168282 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:43.168248    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:48.166269 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.166234    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:48.212200 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.212168    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"]
Sep 17 18:00:48.221478 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:00:48.221445    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:00:48.221478 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:48.221478 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:48.221478 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe" cmd=["/bin/sleep","15"]
Sep 17 18:00:48.221726 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.221489    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" message="" err=<
Sep 17 18:00:48.221726 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:48.221726 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:48.221726 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 17 18:00:48.221726 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:00:48.221503    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:00:48.221726 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:00:48.221726 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:00:48.221726 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" podUID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerName="main" containerID="cri-o://d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe"
Sep 17 18:00:48.221726 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.221533    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" podUID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerName="main" containerID="cri-o://d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe" gracePeriod=60
Sep 17 18:00:48.245893 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.245869    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:48.401594 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.401555    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" podUID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 18:00:48.559120 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.559089    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 18:00:48.624055 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.624021    2581 generic.go:356] "Generic (PLEG): container finished" podID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerID="d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe" exitCode=0
Sep 17 18:00:48.624199 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.624072    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"
Sep 17 18:00:48.624199 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.624100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" event={"ID":"19cb1349-2702-4bd5-af3c-0af3ced5f261","Type":"ContainerDied","Data":"d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe"}
Sep 17 18:00:48.624199 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.624153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j" event={"ID":"19cb1349-2702-4bd5-af3c-0af3ced5f261","Type":"ContainerDied","Data":"f6026929b526bc44d7abb3de4f3d03d834668e08d230dd2f0f06329a40bba426"}
Sep 17 18:00:48.624199 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.624179    2581 scope.go:117] "RemoveContainer" containerID="d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe"
Sep 17 18:00:48.635258 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.634474    2581 scope.go:117] "RemoveContainer" containerID="e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976"
Sep 17 18:00:48.642567 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.642549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cb1349-2702-4bd5-af3c-0af3ced5f261-tls-certs\") pod \"19cb1349-2702-4bd5-af3c-0af3ced5f261\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") "
Sep 17 18:00:48.642668 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.642578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19cb1349-2702-4bd5-af3c-0af3ced5f261-cabundle-cert\") pod \"19cb1349-2702-4bd5-af3c-0af3ced5f261\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") "
Sep 17 18:00:48.642668 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.642636    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6svp7\" (UniqueName: \"kubernetes.io/projected/19cb1349-2702-4bd5-af3c-0af3ced5f261-kube-api-access-6svp7\") pod \"19cb1349-2702-4bd5-af3c-0af3ced5f261\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") "
Sep 17 18:00:48.642668 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.642660    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-model-cache\") pod \"19cb1349-2702-4bd5-af3c-0af3ced5f261\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") "
Sep 17 18:00:48.642828 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.642696    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-kserve-provision-location\") pod \"19cb1349-2702-4bd5-af3c-0af3ced5f261\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") "
Sep 17 18:00:48.642828 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.642721    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-tmp-dir\") pod \"19cb1349-2702-4bd5-af3c-0af3ced5f261\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") "
Sep 17 18:00:48.642828 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.642763    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-home\") pod \"19cb1349-2702-4bd5-af3c-0af3ced5f261\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") "
Sep 17 18:00:48.642828 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.642812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-dshm\") pod \"19cb1349-2702-4bd5-af3c-0af3ced5f261\" (UID: \"19cb1349-2702-4bd5-af3c-0af3ced5f261\") "
Sep 17 18:00:48.644066 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.643099    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-model-cache" (OuterVolumeSpecName: "model-cache") pod "19cb1349-2702-4bd5-af3c-0af3ced5f261" (UID: "19cb1349-2702-4bd5-af3c-0af3ced5f261"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.644066 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.643267    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.644066 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.643376    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19cb1349-2702-4bd5-af3c-0af3ced5f261" (UID: "19cb1349-2702-4bd5-af3c-0af3ced5f261"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.644066 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.643491    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cb1349-2702-4bd5-af3c-0af3ced5f261-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19cb1349-2702-4bd5-af3c-0af3ced5f261" (UID: "19cb1349-2702-4bd5-af3c-0af3ced5f261"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:48.644066 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.643627    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-home" (OuterVolumeSpecName: "home") pod "19cb1349-2702-4bd5-af3c-0af3ced5f261" (UID: "19cb1349-2702-4bd5-af3c-0af3ced5f261"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.644066 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.643952    2581 scope.go:117] "RemoveContainer" containerID="d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe"
Sep 17 18:00:48.646779 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.645221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cb1349-2702-4bd5-af3c-0af3ced5f261-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19cb1349-2702-4bd5-af3c-0af3ced5f261" (UID: "19cb1349-2702-4bd5-af3c-0af3ced5f261"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:48.646779 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:00:48.645300    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe\": container with ID starting with d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe not found: ID does not exist" containerID="d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe"
Sep 17 18:00:48.646779 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.645335    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe"} err="failed to get container status \"d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe\": rpc error: code = NotFound desc = could not find container \"d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe\": container with ID starting with d824f30259fd41c83ea2262edd459b2ca50b3222e9565d3da2fb4910dbb62bbe not found: ID does not exist"
Sep 17 18:00:48.646779 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.645357    2581 scope.go:117] "RemoveContainer" containerID="e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976"
Sep 17 18:00:48.646779 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.645369    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cb1349-2702-4bd5-af3c-0af3ced5f261-kube-api-access-6svp7" (OuterVolumeSpecName: "kube-api-access-6svp7") pod "19cb1349-2702-4bd5-af3c-0af3ced5f261" (UID: "19cb1349-2702-4bd5-af3c-0af3ced5f261"). InnerVolumeSpecName "kube-api-access-6svp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:48.646779 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.645403    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-dshm" (OuterVolumeSpecName: "dshm") pod "19cb1349-2702-4bd5-af3c-0af3ced5f261" (UID: "19cb1349-2702-4bd5-af3c-0af3ced5f261"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.646779 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:00:48.645653    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976\": container with ID starting with e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976 not found: ID does not exist" containerID="e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976"
Sep 17 18:00:48.646779 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.645672    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976"} err="failed to get container status \"e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976\": rpc error: code = NotFound desc = could not find container \"e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976\": container with ID starting with e9c580102051fbb49f153475fe9a738c773d01cc6e377d61d81aa2a2c3f18976 not found: ID does not exist"
Sep 17 18:00:48.672793 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.672769    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19cb1349-2702-4bd5-af3c-0af3ced5f261" (UID: "19cb1349-2702-4bd5-af3c-0af3ced5f261"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:48.744276 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.744243    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6svp7\" (UniqueName: \"kubernetes.io/projected/19cb1349-2702-4bd5-af3c-0af3ced5f261-kube-api-access-6svp7\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.744383 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.744279    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.744383 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.744297    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.744383 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.744313    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.744383 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.744327    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cb1349-2702-4bd5-af3c-0af3ced5f261-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.744383 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.744340    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cb1349-2702-4bd5-af3c-0af3ced5f261-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.744383 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.744352    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19cb1349-2702-4bd5-af3c-0af3ced5f261-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:00:48.954372 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.954338    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"]
Sep 17 18:00:48.961648 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:48.961619    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-cf8fb698d-l2x2j"]
Sep 17 18:00:49.183005 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:49.182913    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19cb1349-2702-4bd5-af3c-0af3ced5f261" path="/var/lib/kubelet/pods/19cb1349-2702-4bd5-af3c-0af3ced5f261/volumes"
Sep 17 18:00:52.953360 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:00:52.953317    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:01:02.953790 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:02.953749    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:01:03.580338 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:03.580307    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:12.954063 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:12.954014    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:01:22.953705 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:22.953667    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:01:32.953388 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:32.953345    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:01:36.548346 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:36.548317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:40.587315 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:40.587275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:42.953660 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:42.953621    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:01:43.274302 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:43.274226    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:49.166256 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:49.166227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:49.262438 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:49.262409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:52.953290 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:52.953251    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:01:55.440995 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.440940    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"]
Sep 17 18:01:55.441493 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.441472    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerName="main"
Sep 17 18:01:55.441536 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.441498    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerName="main"
Sep 17 18:01:55.441536 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.441526    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerName="storage-initializer"
Sep 17 18:01:55.441608 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.441536    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerName="storage-initializer"
Sep 17 18:01:55.441641 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.441634    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="19cb1349-2702-4bd5-af3c-0af3ced5f261" containerName="main"
Sep 17 18:01:55.445305 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.445278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.448156 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.448135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:01:55.448461 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.448440    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 18:01:55.448582 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.448458    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:01:55.449438 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.449398    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:01:55.449615 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.449468    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-twtxs\""
Sep 17 18:01:55.449615 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.449477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:01:55.453231 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.453210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"]
Sep 17 18:01:55.503225 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.503188    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:55.603747 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.603702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.603924 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.603773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-model-cache\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.603924 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.603900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-dshm\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.604048 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.603944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tmp-dir\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.604048 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.604004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxlt\" (UniqueName: \"kubernetes.io/projected/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kube-api-access-6dxlt\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.604048 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.604030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-home\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.604216 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.604085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tls-certs\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.604216 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.604124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705073 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-dshm\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705260 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tmp-dir\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705260 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxlt\" (UniqueName: \"kubernetes.io/projected/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kube-api-access-6dxlt\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705260 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-home\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705260 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tls-certs\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705260 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705260 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705556 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-model-cache\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705609 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tmp-dir\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705664 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-home\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705725 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.705859 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-model-cache\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.706014 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.705991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.707352 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.707333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-dshm\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.708344 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.708323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tls-certs\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.713934 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.713912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxlt\" (UniqueName: \"kubernetes.io/projected/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kube-api-access-6dxlt\") pod \"tokenizer-template-args-test-kserve-756b47dcdd-z94d4\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.756525 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.756497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:01:55.804215 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.803955    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:55.904118 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:55.904085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:56.043033 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:56.043003    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"]
Sep 17 18:01:56.044145 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:01:56.044117    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3862348c_1f9e_4df8_bf15_1c7fcc33ca2e.slice/crio-c70c8754997b0da5ed043544dd5b49fe32d77fd56a705c82f3aca9973d52b2ed WatchSource:0}: Error finding container c70c8754997b0da5ed043544dd5b49fe32d77fd56a705c82f3aca9973d52b2ed: Status 404 returned error can't find the container with id c70c8754997b0da5ed043544dd5b49fe32d77fd56a705c82f3aca9973d52b2ed
Sep 17 18:01:57.015664 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:57.015621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" event={"ID":"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e","Type":"ContainerStarted","Data":"97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e"}
Sep 17 18:01:57.016113 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:57.015735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" event={"ID":"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e","Type":"ContainerStarted","Data":"c70c8754997b0da5ed043544dd5b49fe32d77fd56a705c82f3aca9973d52b2ed"}
Sep 17 18:01:59.029122 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:59.029088    2581 generic.go:356] "Generic (PLEG): container finished" podID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" containerID="97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e" exitCode=0
Sep 17 18:01:59.029488 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:01:59.029162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" event={"ID":"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e","Type":"ContainerDied","Data":"97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e"}
Sep 17 18:02:00.036761 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:00.036723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" event={"ID":"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e","Type":"ContainerStarted","Data":"5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b"}
Sep 17 18:02:00.059128 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:00.059077    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" podStartSLOduration=2.661000884 podStartE2EDuration="5.059062213s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.398061329s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:00.05685891 +0000 UTC m=+1682.981330871" watchObservedRunningTime="2026-09-17 18:02:00.059062213 +0000 UTC m=+1682.983534171"
Sep 17 18:02:00.757054 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:00.757017    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:02:02.953776 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:02.953736    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:02:05.756712 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:05.756680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:02:05.769240 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:05.769221    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:02:06.080347 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:06.080260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:02:12.953581 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:12.953480    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:02:13.072587 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:13.072550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:14.767022 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:14.766962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:22.963110 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:22.963073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:02:22.970786 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:22.970759    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:02:28.517730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:28.517699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:28.530391 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:28.530362    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"]
Sep 17 18:02:28.615018 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:28.614961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:32.278808 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:32.278770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:32.333817 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:32.333797    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:33.049753 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.049724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:33.066819 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.066792    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"]
Sep 17 18:02:33.077903 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:02:33.077870    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:02:33.077903 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:33.077903 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:33.077903 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b" cmd=["/bin/sleep","15"]
Sep 17 18:02:33.078233 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.077922    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" message="" err=<
Sep 17 18:02:33.078233 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:33.078233 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:33.078233 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 17 18:02:33.078233 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:02:33.077934    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:02:33.078233 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:33.078233 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:33.078233 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" podUID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" containerName="main" containerID="cri-o://5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b"
Sep 17 18:02:33.078233 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.077980    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" podUID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" containerName="main" containerID="cri-o://5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b" gracePeriod=60
Sep 17 18:02:33.116055 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.116031    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:33.425427 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.425406    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:02:33.537301 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537271    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-home\") pod \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") "
Sep 17 18:02:33.537476 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537316    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-model-cache\") pod \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") "
Sep 17 18:02:33.537476 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537342    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tmp-dir\") pod \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") "
Sep 17 18:02:33.537476 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537386    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tls-certs\") pod \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") "
Sep 17 18:02:33.537476 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537421    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kserve-provision-location\") pod \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") "
Sep 17 18:02:33.537476 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537473    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-cabundle-cert\") pod \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") "
Sep 17 18:02:33.537755 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dxlt\" (UniqueName: \"kubernetes.io/projected/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kube-api-access-6dxlt\") pod \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") "
Sep 17 18:02:33.537755 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537526    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" (UID: "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.537755 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-home" (OuterVolumeSpecName: "home") pod "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" (UID: "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.537755 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-dshm\") pod \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\" (UID: \"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e\") "
Sep 17 18:02:33.537755 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" (UID: "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.538039 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537875    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.538039 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537895    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.538039 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537910    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.538039 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.537934    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" (UID: "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:33.540322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.540297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-dshm" (OuterVolumeSpecName: "dshm") pod "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" (UID: "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.540448 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.540421    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kube-api-access-6dxlt" (OuterVolumeSpecName: "kube-api-access-6dxlt") pod "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" (UID: "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e"). InnerVolumeSpecName "kube-api-access-6dxlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:33.540506 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.540424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" (UID: "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:33.557904 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.557882    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" (UID: "3862348c-1f9e-4df8-bf15-1c7fcc33ca2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:33.638429 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.638368    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.638429 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.638398    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.638429 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.638415    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.638593 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.638437    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dxlt\" (UniqueName: \"kubernetes.io/projected/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-kube-api-access-6dxlt\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:33.638593 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:33.638452    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:34.245113 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.245078    2581 generic.go:356] "Generic (PLEG): container finished" podID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" containerID="5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b" exitCode=0
Sep 17 18:02:34.245297 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.245135    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"
Sep 17 18:02:34.245297 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.245151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" event={"ID":"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e","Type":"ContainerDied","Data":"5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b"}
Sep 17 18:02:34.245297 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.245199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4" event={"ID":"3862348c-1f9e-4df8-bf15-1c7fcc33ca2e","Type":"ContainerDied","Data":"c70c8754997b0da5ed043544dd5b49fe32d77fd56a705c82f3aca9973d52b2ed"}
Sep 17 18:02:34.245297 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.245221    2581 scope.go:117] "RemoveContainer" containerID="5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b"
Sep 17 18:02:34.254909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.254894    2581 scope.go:117] "RemoveContainer" containerID="97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e"
Sep 17 18:02:34.262770 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.262752    2581 scope.go:117] "RemoveContainer" containerID="5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b"
Sep 17 18:02:34.263051 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:02:34.263033    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b\": container with ID starting with 5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b not found: ID does not exist" containerID="5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b"
Sep 17 18:02:34.263105 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.263058    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b"} err="failed to get container status \"5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b\": rpc error: code = NotFound desc = could not find container \"5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b\": container with ID starting with 5e1ea594d38f13b09e8dbbdc7b1c0eaddc569624858aff2beb5b62cc44f02e0b not found: ID does not exist"
Sep 17 18:02:34.263105 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.263078    2581 scope.go:117] "RemoveContainer" containerID="97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e"
Sep 17 18:02:34.263364 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:02:34.263342    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e\": container with ID starting with 97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e not found: ID does not exist" containerID="97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e"
Sep 17 18:02:34.263459 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.263369    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e"} err="failed to get container status \"97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e\": rpc error: code = NotFound desc = could not find container \"97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e\": container with ID starting with 97a1e73921fa9ed7dfc5d257c77f358c5b23858f02553fbb004804207289246e not found: ID does not exist"
Sep 17 18:02:34.272730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.272712    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"]
Sep 17 18:02:34.277322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:34.277303    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-756b47dcdd-z94d4"]
Sep 17 18:02:35.183130 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:35.183095    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" path="/var/lib/kubelet/pods/3862348c-1f9e-4df8-bf15-1c7fcc33ca2e/volumes"
Sep 17 18:02:39.257702 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:39.257667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:43.554020 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:43.553979    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" containerID="cri-o://6343e6d68972e5ef3566488fd29270821ebe9268e516e0f4e3a6652718a7a566" gracePeriod=45
Sep 17 18:02:43.964568 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:43.964530    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:02:44.105494 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:02:44.105459    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3004b7_a124_481c_90d9_423237d06156.slice/crio-conmon-6343e6d68972e5ef3566488fd29270821ebe9268e516e0f4e3a6652718a7a566.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 18:02:44.299821 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.299792    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f3004b7-a124-481c-90d9-423237d06156" containerID="6343e6d68972e5ef3566488fd29270821ebe9268e516e0f4e3a6652718a7a566" exitCode=0
Sep 17 18:02:44.299937 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.299854    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:02:44.299937 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.299860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" event={"ID":"7f3004b7-a124-481c-90d9-423237d06156","Type":"ContainerDied","Data":"6343e6d68972e5ef3566488fd29270821ebe9268e516e0f4e3a6652718a7a566"}
Sep 17 18:02:44.299937 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.299906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7" event={"ID":"7f3004b7-a124-481c-90d9-423237d06156","Type":"ContainerDied","Data":"2afaa36c4ba052247e9ab61bf59361a426213b50cd70ec1517c2d8fe84197ce3"}
Sep 17 18:02:44.299937 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.299929    2581 scope.go:117] "RemoveContainer" containerID="6343e6d68972e5ef3566488fd29270821ebe9268e516e0f4e3a6652718a7a566"
Sep 17 18:02:44.308310 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.308291    2581 scope.go:117] "RemoveContainer" containerID="7a61b31c74615e94687cc30faf434644e72273e80d6493c4ed6f38d9b59b7621"
Sep 17 18:02:44.416296 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416272    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-dshm\") pod \"7f3004b7-a124-481c-90d9-423237d06156\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") "
Sep 17 18:02:44.416403 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416318    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3004b7-a124-481c-90d9-423237d06156-tls-certs\") pod \"7f3004b7-a124-481c-90d9-423237d06156\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") "
Sep 17 18:02:44.416403 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416347    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-home\") pod \"7f3004b7-a124-481c-90d9-423237d06156\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") "
Sep 17 18:02:44.416403 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3004b7-a124-481c-90d9-423237d06156-cabundle-cert\") pod \"7f3004b7-a124-481c-90d9-423237d06156\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") "
Sep 17 18:02:44.416403 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416390    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-model-cache\") pod \"7f3004b7-a124-481c-90d9-423237d06156\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") "
Sep 17 18:02:44.416626 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-kserve-provision-location\") pod \"7f3004b7-a124-481c-90d9-423237d06156\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") "
Sep 17 18:02:44.416626 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2kxm\" (UniqueName: \"kubernetes.io/projected/7f3004b7-a124-481c-90d9-423237d06156-kube-api-access-v2kxm\") pod \"7f3004b7-a124-481c-90d9-423237d06156\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") "
Sep 17 18:02:44.416626 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-tmp-dir\") pod \"7f3004b7-a124-481c-90d9-423237d06156\" (UID: \"7f3004b7-a124-481c-90d9-423237d06156\") "
Sep 17 18:02:44.416793 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f3004b7-a124-481c-90d9-423237d06156" (UID: "7f3004b7-a124-481c-90d9-423237d06156"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:44.416793 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3004b7-a124-481c-90d9-423237d06156-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f3004b7-a124-481c-90d9-423237d06156" (UID: "7f3004b7-a124-481c-90d9-423237d06156"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:44.416898 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416833    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f3004b7-a124-481c-90d9-423237d06156-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.416898 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.416855    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.417195 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.417164    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-home" (OuterVolumeSpecName: "home") pod "7f3004b7-a124-481c-90d9-423237d06156" (UID: "7f3004b7-a124-481c-90d9-423237d06156"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:44.419123 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.419091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3004b7-a124-481c-90d9-423237d06156-kube-api-access-v2kxm" (OuterVolumeSpecName: "kube-api-access-v2kxm") pod "7f3004b7-a124-481c-90d9-423237d06156" (UID: "7f3004b7-a124-481c-90d9-423237d06156"). InnerVolumeSpecName "kube-api-access-v2kxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:44.419123 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.419099    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3004b7-a124-481c-90d9-423237d06156-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f3004b7-a124-481c-90d9-423237d06156" (UID: "7f3004b7-a124-481c-90d9-423237d06156"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:44.419284 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.419129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-dshm" (OuterVolumeSpecName: "dshm") pod "7f3004b7-a124-481c-90d9-423237d06156" (UID: "7f3004b7-a124-481c-90d9-423237d06156"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:44.429899 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.429876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f3004b7-a124-481c-90d9-423237d06156" (UID: "7f3004b7-a124-481c-90d9-423237d06156"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:44.438823 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.438801    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f3004b7-a124-481c-90d9-423237d06156" (UID: "7f3004b7-a124-481c-90d9-423237d06156"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:44.517849 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.517799    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3004b7-a124-481c-90d9-423237d06156-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.517849 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.517819    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.517849 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.517829    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.517849 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.517837    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2kxm\" (UniqueName: \"kubernetes.io/projected/7f3004b7-a124-481c-90d9-423237d06156-kube-api-access-v2kxm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.517849 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.517847    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.518079 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:44.517856    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3004b7-a124-481c-90d9-423237d06156-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:02:45.305155 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:45.305124    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"
Sep 17 18:02:45.330152 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:45.330129    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"]
Sep 17 18:02:45.334313 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:45.334291    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-849ff9f6d6-lvst7"]
Sep 17 18:02:47.180559 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:02:47.180528    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f3004b7-a124-481c-90d9-423237d06156" path="/var/lib/kubelet/pods/7f3004b7-a124-481c-90d9-423237d06156/volumes"
Sep 17 18:03:18.448304 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:18.448272    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:18.545991 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:18.545937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:18.757096 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:18.757063    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:20.133436 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:20.133398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:22.028090 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:22.028060    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:23.187082 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:23.187053    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:23.277542 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:23.277513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:25.714778 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:25.714732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:30.726232 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:30.726199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:31.920727 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:31.920699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:32.019290 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:32.019263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:34.432025 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:34.431987    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:37.012268 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:37.012235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:37.092772 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:37.092747    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:59.911203 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:59.911173    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:03:59.918907 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:59.918887    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:03:59.949604 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:03:59.949575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.213269 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:04:00.213244    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:16.092306 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:04:16.092272    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:34.054280 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:04:34.054249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:34.121919 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:04:34.121881    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:51.588664 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:04:51.588629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:56.590249 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:04:56.590215    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:56.687875 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:04:56.687844    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:01.652095 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:01.652058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:03.957350 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:03.957314    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:04.325902 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:04.325827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:23.294759 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:23.294683    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:23.422007 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:23.421975    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:25.692780 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:25.692709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:30.032855 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:30.032813    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:31.715146 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:31.715104    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:36.021624 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:36.021591    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:36.104225 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:36.104195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:39.647633 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:39.647592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:47.685187 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:47.685148    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:47.987301 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:47.987264    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:48.090023 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:05:48.089995    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:11.358285 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:11.358257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:11.589850 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:11.589816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:18.932959 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:18.932920    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:54.132312 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132235    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"]
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132578    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="storage-initializer"
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132589    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="storage-initializer"
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132599    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" containerName="storage-initializer"
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132605    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" containerName="storage-initializer"
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132611    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" containerName="main"
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132618    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" containerName="main"
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132640    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main"
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132645    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main"
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132697    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3862348c-1f9e-4df8-bf15-1c7fcc33ca2e" containerName="main"
Sep 17 18:06:54.132765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.132709    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3004b7-a124-481c-90d9-423237d06156" containerName="main"
Sep 17 18:06:54.135703 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.135687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.139609 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.139584    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:06:54.139759 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.139631    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 18:06:54.139759 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.139587    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:06:54.139759 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.139630    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-cj5jk\""
Sep 17 18:06:54.147877 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.147855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"]
Sep 17 18:06:54.173874 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.173848    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:54.225433 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.225409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.225542 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.225473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.225542 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.225534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.225663 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.225561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvrk\" (UniqueName: \"kubernetes.io/projected/9e065dfc-181a-4652-89f6-cbc2808633f0-kube-api-access-pbvrk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.225663 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.225615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.225663 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.225643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e065dfc-181a-4652-89f6-cbc2808633f0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.326838 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.326812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.326989 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.326841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvrk\" (UniqueName: \"kubernetes.io/projected/9e065dfc-181a-4652-89f6-cbc2808633f0-kube-api-access-pbvrk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.326989 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.326873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.326989 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.326890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e065dfc-181a-4652-89f6-cbc2808633f0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.326989 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.326933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.327204 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.326997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.327244 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.327220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.327290 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.327267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.327347 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.327325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.329264 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.329246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.329613 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.329595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e065dfc-181a-4652-89f6-cbc2808633f0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.336342 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.336312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvrk\" (UniqueName: \"kubernetes.io/projected/9e065dfc-181a-4652-89f6-cbc2808633f0-kube-api-access-pbvrk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.446826 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.446797    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:06:54.729725 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:54.729696    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"]
Sep 17 18:06:54.731222 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:06:54.731193    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e065dfc_181a_4652_89f6_cbc2808633f0.slice/crio-b8a7d19e150974474436ac2c7967f983cae15aa4383d4776860343076126d614 WatchSource:0}: Error finding container b8a7d19e150974474436ac2c7967f983cae15aa4383d4776860343076126d614: Status 404 returned error can't find the container with id b8a7d19e150974474436ac2c7967f983cae15aa4383d4776860343076126d614
Sep 17 18:06:55.626629 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:55.626586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4" event={"ID":"9e065dfc-181a-4652-89f6-cbc2808633f0","Type":"ContainerStarted","Data":"bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651"}
Sep 17 18:06:55.627143 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:55.626673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4" event={"ID":"9e065dfc-181a-4652-89f6-cbc2808633f0","Type":"ContainerStarted","Data":"b8a7d19e150974474436ac2c7967f983cae15aa4383d4776860343076126d614"}
Sep 17 18:06:55.642657 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:55.642629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:55.657399 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:55.657340    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4" podStartSLOduration=1.6573205340000001 podStartE2EDuration="1.657320534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:55.655091024 +0000 UTC m=+1978.579562986" watchObservedRunningTime="2026-09-17 18:06:55.657320534 +0000 UTC m=+1978.581792493"
Sep 17 18:06:56.447591 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:06:56.447560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:07:04.121282 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:04.121247    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:04.447145 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:04.447103    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:07:04.459559 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:04.459534    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:07:04.684788 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:04.684761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:07:28.674699 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:28.674667    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"]
Sep 17 18:07:28.683609 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:28.683578    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:28.684412 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:07:28.684381    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:07:28.684412 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:07:28.684412 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:07:28.684412 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651" cmd=["/bin/sleep","15"]
Sep 17 18:07:28.684698 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:28.684429    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4" message="" err=<
Sep 17 18:07:28.684698 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:07:28.684698 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:07:28.684698 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 17 18:07:28.684698 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:07:28.684440    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:07:28.684698 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:07:28.684698 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:07:28.684698 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4" podUID="9e065dfc-181a-4652-89f6-cbc2808633f0" containerName="main" containerID="cri-o://bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651"
Sep 17 18:07:28.684698 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:28.684486    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4" podUID="9e065dfc-181a-4652-89f6-cbc2808633f0" containerName="main" containerID="cri-o://bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651" gracePeriod=60
Sep 17 18:07:28.754893 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:28.754866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:29.025439 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.025417    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:07:29.095535 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.095505    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-dshm\") pod \"9e065dfc-181a-4652-89f6-cbc2808633f0\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") "
Sep 17 18:07:29.095657 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.095610    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e065dfc-181a-4652-89f6-cbc2808633f0-tls-certs\") pod \"9e065dfc-181a-4652-89f6-cbc2808633f0\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") "
Sep 17 18:07:29.095700 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.095672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-tmp-dir\") pod \"9e065dfc-181a-4652-89f6-cbc2808633f0\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") "
Sep 17 18:07:29.095735 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.095699    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-model-cache\") pod \"9e065dfc-181a-4652-89f6-cbc2808633f0\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") "
Sep 17 18:07:29.095776 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.095758    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-home\") pod \"9e065dfc-181a-4652-89f6-cbc2808633f0\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") "
Sep 17 18:07:29.095823 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.095815    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbvrk\" (UniqueName: \"kubernetes.io/projected/9e065dfc-181a-4652-89f6-cbc2808633f0-kube-api-access-pbvrk\") pod \"9e065dfc-181a-4652-89f6-cbc2808633f0\" (UID: \"9e065dfc-181a-4652-89f6-cbc2808633f0\") "
Sep 17 18:07:29.096144 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.095961    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e065dfc-181a-4652-89f6-cbc2808633f0" (UID: "9e065dfc-181a-4652-89f6-cbc2808633f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:29.096144 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.095998    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e065dfc-181a-4652-89f6-cbc2808633f0" (UID: "9e065dfc-181a-4652-89f6-cbc2808633f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:29.096144 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.096092    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-home" (OuterVolumeSpecName: "home") pod "9e065dfc-181a-4652-89f6-cbc2808633f0" (UID: "9e065dfc-181a-4652-89f6-cbc2808633f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:29.096334 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.096148    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:07:29.096334 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.096163    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:07:29.097769 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.097751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e065dfc-181a-4652-89f6-cbc2808633f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e065dfc-181a-4652-89f6-cbc2808633f0" (UID: "9e065dfc-181a-4652-89f6-cbc2808633f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:29.097823 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.097776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-dshm" (OuterVolumeSpecName: "dshm") pod "9e065dfc-181a-4652-89f6-cbc2808633f0" (UID: "9e065dfc-181a-4652-89f6-cbc2808633f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:29.097925 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.097901    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e065dfc-181a-4652-89f6-cbc2808633f0-kube-api-access-pbvrk" (OuterVolumeSpecName: "kube-api-access-pbvrk") pod "9e065dfc-181a-4652-89f6-cbc2808633f0" (UID: "9e065dfc-181a-4652-89f6-cbc2808633f0"). InnerVolumeSpecName "kube-api-access-pbvrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:29.197201 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.197178    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:07:29.197201 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.197201    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbvrk\" (UniqueName: \"kubernetes.io/projected/9e065dfc-181a-4652-89f6-cbc2808633f0-kube-api-access-pbvrk\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:07:29.197336 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.197210    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e065dfc-181a-4652-89f6-cbc2808633f0-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:07:29.197336 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.197219    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e065dfc-181a-4652-89f6-cbc2808633f0-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:07:29.805982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.805938    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e065dfc-181a-4652-89f6-cbc2808633f0" containerID="bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651" exitCode=0
Sep 17 18:07:29.806454 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.806004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4" event={"ID":"9e065dfc-181a-4652-89f6-cbc2808633f0","Type":"ContainerDied","Data":"bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651"}
Sep 17 18:07:29.806454 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.806047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4" event={"ID":"9e065dfc-181a-4652-89f6-cbc2808633f0","Type":"ContainerDied","Data":"b8a7d19e150974474436ac2c7967f983cae15aa4383d4776860343076126d614"}
Sep 17 18:07:29.806454 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.806070    2581 scope.go:117] "RemoveContainer" containerID="bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651"
Sep 17 18:07:29.806454 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.806008    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"
Sep 17 18:07:29.815081 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.815058    2581 scope.go:117] "RemoveContainer" containerID="bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651"
Sep 17 18:07:29.815329 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:07:29.815307    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651\": container with ID starting with bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651 not found: ID does not exist" containerID="bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651"
Sep 17 18:07:29.815402 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.815341    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651"} err="failed to get container status \"bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651\": rpc error: code = NotFound desc = could not find container \"bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651\": container with ID starting with bf9e10b3c8bc28d3b0ae68642be7f26432bf71492db3869cf1ac97df03e14651 not found: ID does not exist"
Sep 17 18:07:29.828054 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.828032    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"]
Sep 17 18:07:29.832892 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:29.832872    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vzfs4"]
Sep 17 18:07:31.180657 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:31.180623    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e065dfc-181a-4652-89f6-cbc2808633f0" path="/var/lib/kubelet/pods/9e065dfc-181a-4652-89f6-cbc2808633f0/volumes"
Sep 17 18:07:38.955546 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:38.955513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:39.083126 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:07:39.083096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:19.019382 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:19.019290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:19.104083 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:19.104051    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:29.037903 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:29.037864    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:29.124488 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:29.124457    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:32.654523 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:32.654486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:32.901985 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:32.901945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:36.638862 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:36.638828    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:42.775288 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:42.775253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:42.874882 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:42.874853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:46.355318 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:46.355289    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:59.958890 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:59.958858    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:08:59.964046 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:08:59.964024    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:09:09.031350 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:09:09.031317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:09.132105 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:09:09.132073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:16.075339 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:09:16.075299    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:22.142558 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:09:22.142516    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:22.243085 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:09:22.243046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:22.743220 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:09:22.743186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:23.216978 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:09:23.216935    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:34.657288 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:09:34.657238    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:46.129564 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:09:46.129485    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:10.636611 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:10.636566    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:15.118338 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.118294    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"]
Sep 17 18:10:15.118758 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.118659    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e065dfc-181a-4652-89f6-cbc2808633f0" containerName="main"
Sep 17 18:10:15.118758 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.118670    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e065dfc-181a-4652-89f6-cbc2808633f0" containerName="main"
Sep 17 18:10:15.118758 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.118733    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e065dfc-181a-4652-89f6-cbc2808633f0" containerName="main"
Sep 17 18:10:15.121681 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.121665    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.124296 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.124273    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 18:10:15.125803 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.125786    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 18:10:15.125884 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.125835    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-qwq9g\""
Sep 17 18:10:15.125934 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.125880    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 18:10:15.131719 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.131697    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"]
Sep 17 18:10:15.184542 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.184509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.184542 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.184543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.184784 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.184578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.184784 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.184655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgq4f\" (UniqueName: \"kubernetes.io/projected/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-kube-api-access-tgq4f\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.184784 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.184757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.184960 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.184800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.198559 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.198538    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:15.285623 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.285591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.285623 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.285625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.285841 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.285666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.285841 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.285817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgq4f\" (UniqueName: \"kubernetes.io/projected/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-kube-api-access-tgq4f\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.285956 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.285895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.286040 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.285952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.286096 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.286075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.286246 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.286226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.286403 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.286373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.288235 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.288215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.288348 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.288334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.294889 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.294867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgq4f\" (UniqueName: \"kubernetes.io/projected/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-kube-api-access-tgq4f\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.432091 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.432025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:15.480254 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.480217    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:15.707241 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:15.707217    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"]
Sep 17 18:10:15.708831 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:10:15.708799    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383f0f74_b9e3_4c14_9f45_fc11ed2f600e.slice/crio-4e80c722a78b79cca0c194ab803b1ee7a2b1cc5ffd70845e96341cb941967f96 WatchSource:0}: Error finding container 4e80c722a78b79cca0c194ab803b1ee7a2b1cc5ffd70845e96341cb941967f96: Status 404 returned error can't find the container with id 4e80c722a78b79cca0c194ab803b1ee7a2b1cc5ffd70845e96341cb941967f96
Sep 17 18:10:16.678024 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:16.677991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s" event={"ID":"383f0f74-b9e3-4c14-9f45-fc11ed2f600e","Type":"ContainerStarted","Data":"ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4"}
Sep 17 18:10:16.678499 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:16.678064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s" event={"ID":"383f0f74-b9e3-4c14-9f45-fc11ed2f600e","Type":"ContainerStarted","Data":"4e80c722a78b79cca0c194ab803b1ee7a2b1cc5ffd70845e96341cb941967f96"}
Sep 17 18:10:16.699044 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:16.699001    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s" podStartSLOduration=1.6989879540000001 podStartE2EDuration="1.698987954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:16.698247538 +0000 UTC m=+2179.622719496" watchObservedRunningTime="2026-09-17 18:10:16.698987954 +0000 UTC m=+2179.623459907"
Sep 17 18:10:17.433065 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:17.433034    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:24.740711 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:24.740673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:25.433144 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:25.433113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:25.445331 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:25.445297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:25.734477 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:25.734451    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:49.651485 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:49.651452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:49.684882 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:49.684853    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"]
Sep 17 18:10:49.693991 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:10:49.693945    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:10:49.693991 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:49.693991 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:49.693991 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4" cmd=["/bin/sleep","15"]
Sep 17 18:10:49.694206 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:49.694007    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s" message="" err=<
Sep 17 18:10:49.694206 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:49.694206 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:49.694206 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 17 18:10:49.694206 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:10:49.694019    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:10:49.694206 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:49.694206 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:49.694206 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s" podUID="383f0f74-b9e3-4c14-9f45-fc11ed2f600e" containerName="main" containerID="cri-o://ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4"
Sep 17 18:10:49.694206 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:49.694061    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s" podUID="383f0f74-b9e3-4c14-9f45-fc11ed2f600e" containerName="main" containerID="cri-o://ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4" gracePeriod=60
Sep 17 18:10:49.723858 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:49.723828    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s" podUID="383f0f74-b9e3-4c14-9f45-fc11ed2f600e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 18:10:49.760625 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:49.760596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:50.043491 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.043464    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:50.052558 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.052536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgq4f\" (UniqueName: \"kubernetes.io/projected/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-kube-api-access-tgq4f\") pod \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") "
Sep 17 18:10:50.052693 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.052582    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-dshm\") pod \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") "
Sep 17 18:10:50.052693 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.052649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-home\") pod \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") "
Sep 17 18:10:50.052693 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.052679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tmp-dir\") pod \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") "
Sep 17 18:10:50.052861 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.052703    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-model-cache\") pod \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") "
Sep 17 18:10:50.052861 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.052771    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tls-certs\") pod \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\" (UID: \"383f0f74-b9e3-4c14-9f45-fc11ed2f600e\") "
Sep 17 18:10:50.052959 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.052900    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-home" (OuterVolumeSpecName: "home") pod "383f0f74-b9e3-4c14-9f45-fc11ed2f600e" (UID: "383f0f74-b9e3-4c14-9f45-fc11ed2f600e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:50.053117 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.053098    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:10:50.053117 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.053102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "383f0f74-b9e3-4c14-9f45-fc11ed2f600e" (UID: "383f0f74-b9e3-4c14-9f45-fc11ed2f600e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:50.053237 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.053165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-model-cache" (OuterVolumeSpecName: "model-cache") pod "383f0f74-b9e3-4c14-9f45-fc11ed2f600e" (UID: "383f0f74-b9e3-4c14-9f45-fc11ed2f600e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:50.055471 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.055444    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "383f0f74-b9e3-4c14-9f45-fc11ed2f600e" (UID: "383f0f74-b9e3-4c14-9f45-fc11ed2f600e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:50.056438 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.056403    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-kube-api-access-tgq4f" (OuterVolumeSpecName: "kube-api-access-tgq4f") pod "383f0f74-b9e3-4c14-9f45-fc11ed2f600e" (UID: "383f0f74-b9e3-4c14-9f45-fc11ed2f600e"). InnerVolumeSpecName "kube-api-access-tgq4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:50.056611 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.056575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-dshm" (OuterVolumeSpecName: "dshm") pod "383f0f74-b9e3-4c14-9f45-fc11ed2f600e" (UID: "383f0f74-b9e3-4c14-9f45-fc11ed2f600e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:50.154466 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.154431    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:10:50.154466 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.154469    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgq4f\" (UniqueName: \"kubernetes.io/projected/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-kube-api-access-tgq4f\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:10:50.154655 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.154484    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:10:50.154655 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.154497    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:10:50.154655 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.154511    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383f0f74-b9e3-4c14-9f45-fc11ed2f600e-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:10:50.864952 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.864918    2581 generic.go:356] "Generic (PLEG): container finished" podID="383f0f74-b9e3-4c14-9f45-fc11ed2f600e" containerID="ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4" exitCode=0
Sep 17 18:10:50.865359 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.864978    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"
Sep 17 18:10:50.865359 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.864993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s" event={"ID":"383f0f74-b9e3-4c14-9f45-fc11ed2f600e","Type":"ContainerDied","Data":"ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4"}
Sep 17 18:10:50.865359 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.865067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s" event={"ID":"383f0f74-b9e3-4c14-9f45-fc11ed2f600e","Type":"ContainerDied","Data":"4e80c722a78b79cca0c194ab803b1ee7a2b1cc5ffd70845e96341cb941967f96"}
Sep 17 18:10:50.865359 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.865090    2581 scope.go:117] "RemoveContainer" containerID="ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4"
Sep 17 18:10:50.874830 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.874811    2581 scope.go:117] "RemoveContainer" containerID="ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4"
Sep 17 18:10:50.875073 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:10:50.875056    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4\": container with ID starting with ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4 not found: ID does not exist" containerID="ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4"
Sep 17 18:10:50.875147 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.875079    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4"} err="failed to get container status \"ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4\": rpc error: code = NotFound desc = could not find container \"ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4\": container with ID starting with ab0b140776714f443b6a3402d8e1884d59b42828c18fece8f1ad4873f43fb7b4 not found: ID does not exist"
Sep 17 18:10:50.896524 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.896503    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"]
Sep 17 18:10:50.907084 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:50.907060    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ckz8s"]
Sep 17 18:10:51.182820 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:10:51.182754    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383f0f74-b9e3-4c14-9f45-fc11ed2f600e" path="/var/lib/kubelet/pods/383f0f74-b9e3-4c14-9f45-fc11ed2f600e/volumes"
Sep 17 18:11:12.215868 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:11:12.215796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:18.116525 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:11:18.116488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:18.189788 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:11:18.189753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:30.534096 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:11:30.534064    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:39.795836 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:11:39.795802    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:39.881495 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:11:39.881459    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:39.951067 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:11:39.951037    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:08.200028 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:08.199994    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:09.860254 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:09.860223    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:14.565990 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:14.565926    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:14.860259 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:14.860163    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:15.083974 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:15.083919    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:20.224338 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:20.224306    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:20.325430 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:20.325380    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:29.907553 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:29.907521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:30.012816 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:30.012788    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:33.957357 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:33.957324    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:36.379444 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:36.379335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:40.826937 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.826856    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"]
Sep 17 18:12:40.827424 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.827264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383f0f74-b9e3-4c14-9f45-fc11ed2f600e" containerName="main"
Sep 17 18:12:40.827424 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.827277    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="383f0f74-b9e3-4c14-9f45-fc11ed2f600e" containerName="main"
Sep 17 18:12:40.827424 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.827346    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="383f0f74-b9e3-4c14-9f45-fc11ed2f600e" containerName="main"
Sep 17 18:12:40.830560 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.830541    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:40.833320 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.833293    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 18:12:40.833441 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.833325    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:12:40.833441 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.833409    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-gpdpt\""
Sep 17 18:12:40.833566 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.833459    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 18:12:40.833624 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.833601    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 18:12:40.833692 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.833672    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 18:12:40.842098 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.842075    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"]
Sep 17 18:12:40.899150 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.899104    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:40.906743 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.906719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:40.906844 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.906752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2441515-8f9e-496b-b6f1-19e948db39d7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:40.906844 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.906770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgb8\" (UniqueName: \"kubernetes.io/projected/e2441515-8f9e-496b-b6f1-19e948db39d7-kube-api-access-szgb8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:40.906983 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.906863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:40.906983 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.906921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:40.906983 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.906957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:40.907173 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.907027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:40.907173 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:40.907093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2441515-8f9e-496b-b6f1-19e948db39d7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.007674 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.007650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2441515-8f9e-496b-b6f1-19e948db39d7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.007776 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.007691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.007776 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.007716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2441515-8f9e-496b-b6f1-19e948db39d7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.007776 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.007740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgb8\" (UniqueName: \"kubernetes.io/projected/e2441515-8f9e-496b-b6f1-19e948db39d7-kube-api-access-szgb8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.007946 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.007776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.007946 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.007923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.008077 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.007980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.008077 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.008024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.008077 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.008060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.008277 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.008255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.008381 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.008344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.008381 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.008375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.008519 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.008413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2441515-8f9e-496b-b6f1-19e948db39d7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.010241 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.010223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.010328 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.010243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2441515-8f9e-496b-b6f1-19e948db39d7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.016079 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.016059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgb8\" (UniqueName: \"kubernetes.io/projected/e2441515-8f9e-496b-b6f1-19e948db39d7-kube-api-access-szgb8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.141892 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.141839    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:41.295335 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.295303    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:41.425791 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.425766    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"]
Sep 17 18:12:41.428164 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:12:41.428135    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2441515_8f9e_496b_b6f1_19e948db39d7.slice/crio-e61e8feaaaee2a1acce8703d0b747e091a3cd6654e0f8902b06ed2ec8d1f6661 WatchSource:0}: Error finding container e61e8feaaaee2a1acce8703d0b747e091a3cd6654e0f8902b06ed2ec8d1f6661: Status 404 returned error can't find the container with id e61e8feaaaee2a1acce8703d0b747e091a3cd6654e0f8902b06ed2ec8d1f6661
Sep 17 18:12:41.446682 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:41.446652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" event={"ID":"e2441515-8f9e-496b-b6f1-19e948db39d7","Type":"ContainerStarted","Data":"e61e8feaaaee2a1acce8703d0b747e091a3cd6654e0f8902b06ed2ec8d1f6661"}
Sep 17 18:12:42.457397 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:42.457353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" event={"ID":"e2441515-8f9e-496b-b6f1-19e948db39d7","Type":"ContainerStarted","Data":"58cda15b41a0553ea12431a11f3e9b726e245153696aec4ccb738144af157a66"}
Sep 17 18:12:42.988566 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:42.988533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:43.092630 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:43.092596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:44.470885 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:44.470794    2581 generic.go:356] "Generic (PLEG): container finished" podID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerID="58cda15b41a0553ea12431a11f3e9b726e245153696aec4ccb738144af157a66" exitCode=0
Sep 17 18:12:44.470885 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:44.470875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" event={"ID":"e2441515-8f9e-496b-b6f1-19e948db39d7","Type":"ContainerDied","Data":"58cda15b41a0553ea12431a11f3e9b726e245153696aec4ccb738144af157a66"}
Sep 17 18:12:45.478914 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:45.478882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" event={"ID":"e2441515-8f9e-496b-b6f1-19e948db39d7","Type":"ContainerStarted","Data":"33c4e360aa9b65145d5f66910cb542bb3b5c8fde0669c866d07713c3e5c3e341"}
Sep 17 18:12:45.503538 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:45.503490    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podStartSLOduration=2.773125669 podStartE2EDuration="5.5034745s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.730348831s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:45.500604013 +0000 UTC m=+2328.425075972" watchObservedRunningTime="2026-09-17 18:12:45.5034745 +0000 UTC m=+2328.427946459"
Sep 17 18:12:46.143051 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:46.143017    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:51.142931 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:51.142888    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:12:51.144578 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:51.144549    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 18:12:52.830413 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:12:52.830377    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:01.142249 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:13:01.142205    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 18:13:11.142518 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:13:11.142472    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 18:13:21.143035 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:13:21.142987    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 18:13:31.142362 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:13:31.142312    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 18:13:41.142815 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:13:41.142766    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 18:13:42.806125 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:13:42.806014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:51.142193 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:13:51.142157    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 18:13:59.995815 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:13:59.995784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:14:00.003248 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:00.003227    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:14:01.153177 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:01.153142    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:14:01.160867 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:01.160838    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:14:20.457171 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:20.457081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:20.524468 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:20.524410    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:26.569487 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:26.569456    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"]
Sep 17 18:14:26.571549 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:26.571512    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:26.632122 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:26.632096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:41.591382 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:41.591320    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main" containerID="cri-o://33c4e360aa9b65145d5f66910cb542bb3b5c8fde0669c866d07713c3e5c3e341" gracePeriod=45
Sep 17 18:14:42.146103 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.146074    2581 generic.go:356] "Generic (PLEG): container finished" podID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerID="33c4e360aa9b65145d5f66910cb542bb3b5c8fde0669c866d07713c3e5c3e341" exitCode=0
Sep 17 18:14:42.146250 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.146150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" event={"ID":"e2441515-8f9e-496b-b6f1-19e948db39d7","Type":"ContainerDied","Data":"33c4e360aa9b65145d5f66910cb542bb3b5c8fde0669c866d07713c3e5c3e341"}
Sep 17 18:14:42.154447 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.154420    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 18:14:42.232482 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.232460    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:14:42.357168 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357104    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-kserve-provision-location\") pod \"e2441515-8f9e-496b-b6f1-19e948db39d7\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") "
Sep 17 18:14:42.357168 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357134    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-dshm\") pod \"e2441515-8f9e-496b-b6f1-19e948db39d7\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") "
Sep 17 18:14:42.357168 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357152    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szgb8\" (UniqueName: \"kubernetes.io/projected/e2441515-8f9e-496b-b6f1-19e948db39d7-kube-api-access-szgb8\") pod \"e2441515-8f9e-496b-b6f1-19e948db39d7\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") "
Sep 17 18:14:42.357168 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357168    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-home\") pod \"e2441515-8f9e-496b-b6f1-19e948db39d7\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") "
Sep 17 18:14:42.357457 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357185    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-tmp-dir\") pod \"e2441515-8f9e-496b-b6f1-19e948db39d7\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") "
Sep 17 18:14:42.357457 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2441515-8f9e-496b-b6f1-19e948db39d7-tls-certs\") pod \"e2441515-8f9e-496b-b6f1-19e948db39d7\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") "
Sep 17 18:14:42.357457 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357240    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2441515-8f9e-496b-b6f1-19e948db39d7-cabundle-cert\") pod \"e2441515-8f9e-496b-b6f1-19e948db39d7\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") "
Sep 17 18:14:42.357457 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357305    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-model-cache\") pod \"e2441515-8f9e-496b-b6f1-19e948db39d7\" (UID: \"e2441515-8f9e-496b-b6f1-19e948db39d7\") "
Sep 17 18:14:42.357675 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-home" (OuterVolumeSpecName: "home") pod "e2441515-8f9e-496b-b6f1-19e948db39d7" (UID: "e2441515-8f9e-496b-b6f1-19e948db39d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:42.357675 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357616    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:14:42.357789 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357754    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2441515-8f9e-496b-b6f1-19e948db39d7" (UID: "e2441515-8f9e-496b-b6f1-19e948db39d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:42.357879 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.357840    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2441515-8f9e-496b-b6f1-19e948db39d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2441515-8f9e-496b-b6f1-19e948db39d7" (UID: "e2441515-8f9e-496b-b6f1-19e948db39d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:42.359697 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.359673    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2441515-8f9e-496b-b6f1-19e948db39d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2441515-8f9e-496b-b6f1-19e948db39d7" (UID: "e2441515-8f9e-496b-b6f1-19e948db39d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:42.359865 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.359843    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-dshm" (OuterVolumeSpecName: "dshm") pod "e2441515-8f9e-496b-b6f1-19e948db39d7" (UID: "e2441515-8f9e-496b-b6f1-19e948db39d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:42.359865 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.359857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2441515-8f9e-496b-b6f1-19e948db39d7-kube-api-access-szgb8" (OuterVolumeSpecName: "kube-api-access-szgb8") pod "e2441515-8f9e-496b-b6f1-19e948db39d7" (UID: "e2441515-8f9e-496b-b6f1-19e948db39d7"). InnerVolumeSpecName "kube-api-access-szgb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:42.374901 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.374876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2441515-8f9e-496b-b6f1-19e948db39d7" (UID: "e2441515-8f9e-496b-b6f1-19e948db39d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:42.386068 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.386042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2441515-8f9e-496b-b6f1-19e948db39d7" (UID: "e2441515-8f9e-496b-b6f1-19e948db39d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:42.458563 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.458539    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:14:42.458563 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.458560    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:14:42.458677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.458570    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgb8\" (UniqueName: \"kubernetes.io/projected/e2441515-8f9e-496b-b6f1-19e948db39d7-kube-api-access-szgb8\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:14:42.458677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.458580    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:14:42.458677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.458588    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2441515-8f9e-496b-b6f1-19e948db39d7-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:14:42.458677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.458596    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2441515-8f9e-496b-b6f1-19e948db39d7-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:14:42.458677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:42.458603    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2441515-8f9e-496b-b6f1-19e948db39d7-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:14:43.154371 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:43.154336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"
Sep 17 18:14:43.154371 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:43.154344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg" event={"ID":"e2441515-8f9e-496b-b6f1-19e948db39d7","Type":"ContainerDied","Data":"e61e8feaaaee2a1acce8703d0b747e091a3cd6654e0f8902b06ed2ec8d1f6661"}
Sep 17 18:14:43.154922 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:43.154429    2581 scope.go:117] "RemoveContainer" containerID="33c4e360aa9b65145d5f66910cb542bb3b5c8fde0669c866d07713c3e5c3e341"
Sep 17 18:14:43.174107 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:43.174080    2581 scope.go:117] "RemoveContainer" containerID="58cda15b41a0553ea12431a11f3e9b726e245153696aec4ccb738144af157a66"
Sep 17 18:14:43.186706 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:43.186680    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"]
Sep 17 18:14:43.190267 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:43.190242    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7688b48f7c-jt9zg"]
Sep 17 18:14:45.180546 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:14:45.180504    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" path="/var/lib/kubelet/pods/e2441515-8f9e-496b-b6f1-19e948db39d7/volumes"
Sep 17 18:15:10.576427 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:10.576397    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:10.675824 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:10.675797    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:10.814553 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:10.814531    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:10.918010 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:10.917911    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:14.082157 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:14.082128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:16.726154 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:16.726125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:16.825801 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:16.825770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:17.751269 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:17.751231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:23.138690 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:23.138648    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:23.747846 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:23.747816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:23.848361 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:23.848292    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:28.926559 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:28.926519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:29.026303 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:29.026272    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:37.861395 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:37.861347    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:38.498710 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:38.498675    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:38.602036 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:38.602008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:57.899909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.899815    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"]
Sep 17 18:15:57.900485 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.900395    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main"
Sep 17 18:15:57.900485 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.900418    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main"
Sep 17 18:15:57.900485 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.900437    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="storage-initializer"
Sep 17 18:15:57.900485 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.900446    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="storage-initializer"
Sep 17 18:15:57.900727 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.900543    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2441515-8f9e-496b-b6f1-19e948db39d7" containerName="main"
Sep 17 18:15:57.903796 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.903773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:57.906502 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.906478    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 18:15:57.906622 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.906559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 18:15:57.906622 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.906566    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:15:57.906746 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.906657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-dthlj\""
Sep 17 18:15:57.906746 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.906715    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 18:15:57.906746 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.906741    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 18:15:57.915331 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.915308    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"]
Sep 17 18:15:57.940012 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:57.939984    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:58.026328 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.026305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbmr\" (UniqueName: \"kubernetes.io/projected/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kube-api-access-clbmr\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.026433 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.026350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.026433 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.026370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.026523 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.026458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.026523 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.026484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.026632 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.026526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.026632 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.026546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.026632 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.026603    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/191dac7a-bc0c-47ca-9c85-77d4617f67bd-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127085 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127235 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127235 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127235 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127394 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/191dac7a-bc0c-47ca-9c85-77d4617f67bd-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127394 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbmr\" (UniqueName: \"kubernetes.io/projected/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kube-api-access-clbmr\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127394 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127394 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127652 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127834 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.127987 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.127727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.128201 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.128173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.128356 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.128306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/191dac7a-bc0c-47ca-9c85-77d4617f67bd-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.130066 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.130028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.130931 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.130881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.139977 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.139933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbmr\" (UniqueName: \"kubernetes.io/projected/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kube-api-access-clbmr\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.156228 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.156172    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:58.216214 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.216184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:15:58.499539 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.499510    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"]
Sep 17 18:15:58.501649 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:15:58.501619    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191dac7a_bc0c_47ca_9c85_77d4617f67bd.slice/crio-e1391a617ec1fe58cda741c351f38eb7bafc509d77a176ab9dfa4670320bcbdf WatchSource:0}: Error finding container e1391a617ec1fe58cda741c351f38eb7bafc509d77a176ab9dfa4670320bcbdf: Status 404 returned error can't find the container with id e1391a617ec1fe58cda741c351f38eb7bafc509d77a176ab9dfa4670320bcbdf
Sep 17 18:15:58.546727 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:58.546699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" event={"ID":"191dac7a-bc0c-47ca-9c85-77d4617f67bd","Type":"ContainerStarted","Data":"e1391a617ec1fe58cda741c351f38eb7bafc509d77a176ab9dfa4670320bcbdf"}
Sep 17 18:15:59.558213 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:15:59.558179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" event={"ID":"191dac7a-bc0c-47ca-9c85-77d4617f67bd","Type":"ContainerStarted","Data":"a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9"}
Sep 17 18:16:02.576171 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:02.576136    2581 generic.go:356] "Generic (PLEG): container finished" podID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerID="a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9" exitCode=0
Sep 17 18:16:02.576596 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:02.576201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" event={"ID":"191dac7a-bc0c-47ca-9c85-77d4617f67bd","Type":"ContainerDied","Data":"a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9"}
Sep 17 18:16:03.584862 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:03.584822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" event={"ID":"191dac7a-bc0c-47ca-9c85-77d4617f67bd","Type":"ContainerStarted","Data":"f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78"}
Sep 17 18:16:03.609669 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:03.609617    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" podStartSLOduration=3.553376795 podStartE2EDuration="6.609602733s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.056225938s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:03.607504611 +0000 UTC m=+2526.531976570" watchObservedRunningTime="2026-09-17 18:16:03.609602733 +0000 UTC m=+2526.534074692"
Sep 17 18:16:04.216777 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:04.216734    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:16:08.216549 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:08.216513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:16:08.218048 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:08.218022    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 17 18:16:15.914738 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:15.914697    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:18.217463 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:18.217422    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 17 18:16:28.217596 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:28.217556    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 17 18:16:38.217637 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:38.217586    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 17 18:16:43.335043 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:43.335011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:48.217042 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:48.217004    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 17 18:16:58.216609 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:16:58.216567    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 17 18:17:05.824060 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:05.824024    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:08.217560 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:08.217517    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 17 18:17:17.084438 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:17.084360    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:18.227655 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:18.227618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:17:18.235688 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:18.235655    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:17:20.206581 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:20.206544    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:20.214637 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:20.214609    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"]
Sep 17 18:17:20.266004 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:20.265957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:35.243792 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.243750    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main" containerID="cri-o://f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78" gracePeriod=45
Sep 17 18:17:35.794014 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.793991    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:17:35.835101 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835066    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-model-cache\") pod \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") "
Sep 17 18:17:35.835236 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835122    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tls-certs\") pod \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") "
Sep 17 18:17:35.835236 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835157    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-dshm\") pod \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") "
Sep 17 18:17:35.835236 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835226    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clbmr\" (UniqueName: \"kubernetes.io/projected/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kube-api-access-clbmr\") pod \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") "
Sep 17 18:17:35.835407 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835252    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-home\") pod \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") "
Sep 17 18:17:35.835407 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "191dac7a-bc0c-47ca-9c85-77d4617f67bd" (UID: "191dac7a-bc0c-47ca-9c85-77d4617f67bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:35.835407 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835302    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kserve-provision-location\") pod \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") "
Sep 17 18:17:35.835407 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tmp-dir\") pod \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") "
Sep 17 18:17:35.835407 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/191dac7a-bc0c-47ca-9c85-77d4617f67bd-cabundle-cert\") pod \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\" (UID: \"191dac7a-bc0c-47ca-9c85-77d4617f67bd\") "
Sep 17 18:17:35.835648 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835629    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:17:35.835648 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-home" (OuterVolumeSpecName: "home") pod "191dac7a-bc0c-47ca-9c85-77d4617f67bd" (UID: "191dac7a-bc0c-47ca-9c85-77d4617f67bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:35.835947 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.835929    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191dac7a-bc0c-47ca-9c85-77d4617f67bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "191dac7a-bc0c-47ca-9c85-77d4617f67bd" (UID: "191dac7a-bc0c-47ca-9c85-77d4617f67bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:35.836068 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.836041    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "191dac7a-bc0c-47ca-9c85-77d4617f67bd" (UID: "191dac7a-bc0c-47ca-9c85-77d4617f67bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:35.837870 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.837839    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "191dac7a-bc0c-47ca-9c85-77d4617f67bd" (UID: "191dac7a-bc0c-47ca-9c85-77d4617f67bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:35.837870 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.837862    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-dshm" (OuterVolumeSpecName: "dshm") pod "191dac7a-bc0c-47ca-9c85-77d4617f67bd" (UID: "191dac7a-bc0c-47ca-9c85-77d4617f67bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:35.838012 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.837941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kube-api-access-clbmr" (OuterVolumeSpecName: "kube-api-access-clbmr") pod "191dac7a-bc0c-47ca-9c85-77d4617f67bd" (UID: "191dac7a-bc0c-47ca-9c85-77d4617f67bd"). InnerVolumeSpecName "kube-api-access-clbmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:35.859333 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.859281    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "191dac7a-bc0c-47ca-9c85-77d4617f67bd" (UID: "191dac7a-bc0c-47ca-9c85-77d4617f67bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:35.936232 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.936211    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:17:35.936232 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.936231    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:17:35.936384 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.936240    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clbmr\" (UniqueName: \"kubernetes.io/projected/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kube-api-access-clbmr\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:17:35.936384 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.936249    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:17:35.936384 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.936259    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:17:35.936384 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.936285    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/191dac7a-bc0c-47ca-9c85-77d4617f67bd-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:17:35.936384 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:35.936294    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/191dac7a-bc0c-47ca-9c85-77d4617f67bd-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:17:36.108010 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.107980    2581 generic.go:356] "Generic (PLEG): container finished" podID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerID="f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78" exitCode=0
Sep 17 18:17:36.108110 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.108037    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"
Sep 17 18:17:36.108110 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.108042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" event={"ID":"191dac7a-bc0c-47ca-9c85-77d4617f67bd","Type":"ContainerDied","Data":"f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78"}
Sep 17 18:17:36.108201 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.108117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv" event={"ID":"191dac7a-bc0c-47ca-9c85-77d4617f67bd","Type":"ContainerDied","Data":"e1391a617ec1fe58cda741c351f38eb7bafc509d77a176ab9dfa4670320bcbdf"}
Sep 17 18:17:36.108201 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.108142    2581 scope.go:117] "RemoveContainer" containerID="f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78"
Sep 17 18:17:36.118381 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.118362    2581 scope.go:117] "RemoveContainer" containerID="a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9"
Sep 17 18:17:36.127677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.127657    2581 scope.go:117] "RemoveContainer" containerID="f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78"
Sep 17 18:17:36.128037 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:17:36.128008    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78\": container with ID starting with f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78 not found: ID does not exist" containerID="f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78"
Sep 17 18:17:36.128106 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.128052    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78"} err="failed to get container status \"f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78\": rpc error: code = NotFound desc = could not find container \"f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78\": container with ID starting with f8419d131f64d7dfb4370f8998251a95d462b40d8de058c40dae6a331df95c78 not found: ID does not exist"
Sep 17 18:17:36.128106 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.128079    2581 scope.go:117] "RemoveContainer" containerID="a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9"
Sep 17 18:17:36.128337 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:17:36.128317    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9\": container with ID starting with a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9 not found: ID does not exist" containerID="a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9"
Sep 17 18:17:36.128395 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.128347    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9"} err="failed to get container status \"a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9\": rpc error: code = NotFound desc = could not find container \"a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9\": container with ID starting with a53309abf825f3b5f1d7208745e4bac7c4079900558e335c2bdc075830c8b0e9 not found: ID does not exist"
Sep 17 18:17:36.132714 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.132687    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"]
Sep 17 18:17:36.137297 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:36.137275    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78dc4f76fd-xv9wv"]
Sep 17 18:17:37.180129 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:37.180098    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" path="/var/lib/kubelet/pods/191dac7a-bc0c-47ca-9c85-77d4617f67bd/volumes"
Sep 17 18:17:42.988583 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:42.988547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:43.047351 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:43.047323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:47.269394 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:47.269362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:47.370101 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:17:47.370076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:09.438154 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:09.438115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:09.534228 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:09.534195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:16.116398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:16.116364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:21.961737 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:21.961700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:22.061128 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:22.061094    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:25.869707 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:25.869672    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:26.073794 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:26.073758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:33.006828 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:33.006793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:33.105065 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:33.105000    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:33.200112 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:33.200080    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:38.102428 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:38.102377    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:39.561874 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.561837    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"]
Sep 17 18:18:39.562301 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.562213    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="storage-initializer"
Sep 17 18:18:39.562301 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.562225    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="storage-initializer"
Sep 17 18:18:39.562301 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.562235    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main"
Sep 17 18:18:39.562301 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.562241    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main"
Sep 17 18:18:39.562301 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.562300    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="191dac7a-bc0c-47ca-9c85-77d4617f67bd" containerName="main"
Sep 17 18:18:39.565677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.565656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.568454 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.568412    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 18:18:39.568454 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.568421    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-gfk5w\""
Sep 17 18:18:39.568652 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.568608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 18:18:39.568706 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.568685    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-dpg7g\""
Sep 17 18:18:39.569409 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.569387    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:18:39.569523 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.569409    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 18:18:39.569523 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.569461    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 18:18:39.578086 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.578041    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"]
Sep 17 18:18:39.582287 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.582267    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.582860 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.582838    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"]
Sep 17 18:18:39.595688 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.595665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"]
Sep 17 18:18:39.623887 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.623859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:39.634058 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.634145 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.634145 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rc22\" (UniqueName: \"kubernetes.io/projected/887a5a95-46ea-4df2-9686-b56055210ac8-kube-api-access-5rc22\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.634218 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.634218 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.634218 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/957cd07b-8af6-4187-bb39-0a48d954d3e9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.634315 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957cd07b-8af6-4187-bb39-0a48d954d3e9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.634315 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.634396 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.634396 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.634461 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/887a5a95-46ea-4df2-9686-b56055210ac8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.634461 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nshp5\" (UniqueName: \"kubernetes.io/projected/957cd07b-8af6-4187-bb39-0a48d954d3e9-kube-api-access-nshp5\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.634461 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.634560 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.634560 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887a5a95-46ea-4df2-9686-b56055210ac8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.634560 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.634528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.735580 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957cd07b-8af6-4187-bb39-0a48d954d3e9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.735580 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/887a5a95-46ea-4df2-9686-b56055210ac8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nshp5\" (UniqueName: \"kubernetes.io/projected/957cd07b-8af6-4187-bb39-0a48d954d3e9-kube-api-access-nshp5\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887a5a95-46ea-4df2-9686-b56055210ac8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.735832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.736398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rc22\" (UniqueName: \"kubernetes.io/projected/887a5a95-46ea-4df2-9686-b56055210ac8-kube-api-access-5rc22\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.736398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.736398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.736398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.735907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/957cd07b-8af6-4187-bb39-0a48d954d3e9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.736398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.736084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.736398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.736105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.736398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.736085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.736398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.736322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.736821 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.736577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.736821 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.736620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.736821 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.736768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/887a5a95-46ea-4df2-9686-b56055210ac8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.737011 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.736908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.737011 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.736993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.737130 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.737025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/957cd07b-8af6-4187-bb39-0a48d954d3e9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.738716 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.738692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957cd07b-8af6-4187-bb39-0a48d954d3e9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.738834 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.738753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887a5a95-46ea-4df2-9686-b56055210ac8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.738834 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.738789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.738943 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.738924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.744688 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.744656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nshp5\" (UniqueName: \"kubernetes.io/projected/957cd07b-8af6-4187-bb39-0a48d954d3e9-kube-api-access-nshp5\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.744785 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.744722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rc22\" (UniqueName: \"kubernetes.io/projected/887a5a95-46ea-4df2-9686-b56055210ac8-kube-api-access-5rc22\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:39.831024 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.830925    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:39.876115 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.876092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:39.893740 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:39.893718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:40.159845 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:40.159814    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"]
Sep 17 18:18:40.161390 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:18:40.161359    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957cd07b_8af6_4187_bb39_0a48d954d3e9.slice/crio-d32901bdeddad2018ecd8792b53f0d115a46d64f67ebc409ccf89c811e726321 WatchSource:0}: Error finding container d32901bdeddad2018ecd8792b53f0d115a46d64f67ebc409ccf89c811e726321: Status 404 returned error can't find the container with id d32901bdeddad2018ecd8792b53f0d115a46d64f67ebc409ccf89c811e726321
Sep 17 18:18:40.163522 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:40.163502    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:18:40.186012 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:40.185983    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"]
Sep 17 18:18:40.187682 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:18:40.187661    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887a5a95_46ea_4df2_9686_b56055210ac8.slice/crio-4f2f4c896d3ae6beb8d1da2f51d6d62e7dc2404b5f71b23c75ffef48122b747b WatchSource:0}: Error finding container 4f2f4c896d3ae6beb8d1da2f51d6d62e7dc2404b5f71b23c75ffef48122b747b: Status 404 returned error can't find the container with id 4f2f4c896d3ae6beb8d1da2f51d6d62e7dc2404b5f71b23c75ffef48122b747b
Sep 17 18:18:40.459016 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:40.458983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" event={"ID":"957cd07b-8af6-4187-bb39-0a48d954d3e9","Type":"ContainerStarted","Data":"d32901bdeddad2018ecd8792b53f0d115a46d64f67ebc409ccf89c811e726321"}
Sep 17 18:18:40.460613 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:40.460586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" event={"ID":"887a5a95-46ea-4df2-9686-b56055210ac8","Type":"ContainerStarted","Data":"9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445"}
Sep 17 18:18:40.460728 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:40.460629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" event={"ID":"887a5a95-46ea-4df2-9686-b56055210ac8","Type":"ContainerStarted","Data":"4f2f4c896d3ae6beb8d1da2f51d6d62e7dc2404b5f71b23c75ffef48122b747b"}
Sep 17 18:18:42.472533 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:42.472455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" event={"ID":"957cd07b-8af6-4187-bb39-0a48d954d3e9","Type":"ContainerStarted","Data":"cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25"}
Sep 17 18:18:42.472905 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:42.472599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:43.478835 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:43.478803    2581 generic.go:356] "Generic (PLEG): container finished" podID="887a5a95-46ea-4df2-9686-b56055210ac8" containerID="9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445" exitCode=0
Sep 17 18:18:43.479352 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:43.478878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" event={"ID":"887a5a95-46ea-4df2-9686-b56055210ac8","Type":"ContainerDied","Data":"9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445"}
Sep 17 18:18:43.481001 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:43.480952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" event={"ID":"957cd07b-8af6-4187-bb39-0a48d954d3e9","Type":"ContainerStarted","Data":"fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9"}
Sep 17 18:18:44.488152 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:44.488118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" event={"ID":"887a5a95-46ea-4df2-9686-b56055210ac8","Type":"ContainerStarted","Data":"ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773"}
Sep 17 18:18:44.511145 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:44.511087    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podStartSLOduration=3.124377224 podStartE2EDuration="5.511065982s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.386688758s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:44.509490981 +0000 UTC m=+2687.433962951" watchObservedRunningTime="2026-09-17 18:18:44.511065982 +0000 UTC m=+2687.435537943"
Sep 17 18:18:45.495911 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:45.495877    2581 generic.go:356] "Generic (PLEG): container finished" podID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerID="fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9" exitCode=0
Sep 17 18:18:45.496320 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:45.495924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" event={"ID":"957cd07b-8af6-4187-bb39-0a48d954d3e9","Type":"ContainerDied","Data":"fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9"}
Sep 17 18:18:46.506410 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:46.506365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" event={"ID":"957cd07b-8af6-4187-bb39-0a48d954d3e9","Type":"ContainerStarted","Data":"e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784"}
Sep 17 18:18:46.532573 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:46.532513    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podStartSLOduration=3.173364621 podStartE2EDuration="7.532499213s" totalImagesPullingTime="2.054894121s" totalInitContainerRuntime="2.304240471s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:46.530625537 +0000 UTC m=+2689.455097500" watchObservedRunningTime="2026-09-17 18:18:46.532499213 +0000 UTC m=+2689.456971171"
Sep 17 18:18:49.526814 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:49.526776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:49.876953 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:49.876858    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:49.876953 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:49.876929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:18:49.878191 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:49.878154    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:18:49.893945 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:49.893916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:49.894085 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:49.894012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:18:49.895171 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:49.895142    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:18:59.729820 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:59.729760    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:59.877346 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:59.877300    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:18:59.894181 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:59.894141    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:18:59.894628 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:18:59.894607    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:19:00.042253 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:00.042160    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:19:00.049680 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:00.049659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:19:00.522610 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:00.522580    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:06.471106 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:06.471072    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:09.877331 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:09.877284    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:19:09.894672 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:09.894628    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:19:19.876975 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:19.876915    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:19:19.894643 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:19.894607    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:19:20.112109 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:20.112081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:21.122474 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:21.122445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:29.786181 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:29.786146    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:29.876794 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:29.876749    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:19:29.886348 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:29.886318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:29.894850 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:29.894815    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:19:39.876810 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:39.876759    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:19:39.894786 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:39.894749    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:19:49.877224 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:49.877177    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:19:49.894126 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:49.894089    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:19:59.877458 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:59.877414    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:19:59.894736 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:19:59.894704    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:20:03.281134 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:03.281086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:03.397827 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:03.397805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:09.877456 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:09.877405    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:20:09.895063 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:09.895025    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:20:19.877682 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:19.877587    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:20:19.894217 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:19.894176    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:20:27.894876 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:27.894826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:27.995952 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:27.995918    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:29.876766 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:29.876727    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:20:29.894514 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:29.894468    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:20:39.877088 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:39.877037    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:20:39.894785 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:39.894753    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:20:49.877292 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:49.877243    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:20:49.895137 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:49.895100    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:20:53.854039 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:53.853991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:54.152126 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:54.152036    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:59.877407 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:59.877358    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:20:59.895000 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:20:59.894945    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:21:08.735916 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:08.735880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:09.877424 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:09.877378    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 18:21:09.894763 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:09.894713    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 17 18:21:19.886952 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:19.886921    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:21:19.903585 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:19.903560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:21:19.905780 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:19.905758    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:21:19.914526 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:19.914503    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:21:29.065755 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:29.065721    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:29.067742 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:29.067718    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"]
Sep 17 18:21:29.073371 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:29.073344    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"]
Sep 17 18:21:29.126707 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:29.126679    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:44.091176 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:44.091058    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main" containerID="cri-o://ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773" gracePeriod=45
Sep 17 18:21:44.096179 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:44.096131    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main" containerID="cri-o://e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784" gracePeriod=45
Sep 17 18:21:44.912316 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:44.912282    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="llm-d-routing-sidecar" containerID="cri-o://cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25" gracePeriod=45
Sep 17 18:21:45.038769 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.038749    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:21:45.134097 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134070    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-tmp-dir\") pod \"887a5a95-46ea-4df2-9686-b56055210ac8\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") "
Sep 17 18:21:45.134445 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-model-cache\") pod \"887a5a95-46ea-4df2-9686-b56055210ac8\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") "
Sep 17 18:21:45.134445 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/887a5a95-46ea-4df2-9686-b56055210ac8-cabundle-cert\") pod \"887a5a95-46ea-4df2-9686-b56055210ac8\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") "
Sep 17 18:21:45.134445 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134202    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-home\") pod \"887a5a95-46ea-4df2-9686-b56055210ac8\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") "
Sep 17 18:21:45.134445 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134220    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-dshm\") pod \"887a5a95-46ea-4df2-9686-b56055210ac8\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") "
Sep 17 18:21:45.134445 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134246    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-kserve-provision-location\") pod \"887a5a95-46ea-4df2-9686-b56055210ac8\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") "
Sep 17 18:21:45.134445 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134286    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887a5a95-46ea-4df2-9686-b56055210ac8-tls-certs\") pod \"887a5a95-46ea-4df2-9686-b56055210ac8\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") "
Sep 17 18:21:45.134445 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134321    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rc22\" (UniqueName: \"kubernetes.io/projected/887a5a95-46ea-4df2-9686-b56055210ac8-kube-api-access-5rc22\") pod \"887a5a95-46ea-4df2-9686-b56055210ac8\" (UID: \"887a5a95-46ea-4df2-9686-b56055210ac8\") "
Sep 17 18:21:45.134851 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134538    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887a5a95-46ea-4df2-9686-b56055210ac8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "887a5a95-46ea-4df2-9686-b56055210ac8" (UID: "887a5a95-46ea-4df2-9686-b56055210ac8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:45.135005 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.134982    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/887a5a95-46ea-4df2-9686-b56055210ac8-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.135108 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.135008    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-home" (OuterVolumeSpecName: "home") pod "887a5a95-46ea-4df2-9686-b56055210ac8" (UID: "887a5a95-46ea-4df2-9686-b56055210ac8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.136125 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.136095    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-model-cache" (OuterVolumeSpecName: "model-cache") pod "887a5a95-46ea-4df2-9686-b56055210ac8" (UID: "887a5a95-46ea-4df2-9686-b56055210ac8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.138025 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.137999    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-dshm" (OuterVolumeSpecName: "dshm") pod "887a5a95-46ea-4df2-9686-b56055210ac8" (UID: "887a5a95-46ea-4df2-9686-b56055210ac8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.138161 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.138143    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887a5a95-46ea-4df2-9686-b56055210ac8-kube-api-access-5rc22" (OuterVolumeSpecName: "kube-api-access-5rc22") pod "887a5a95-46ea-4df2-9686-b56055210ac8" (UID: "887a5a95-46ea-4df2-9686-b56055210ac8"). InnerVolumeSpecName "kube-api-access-5rc22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:45.138222 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.138167    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887a5a95-46ea-4df2-9686-b56055210ac8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "887a5a95-46ea-4df2-9686-b56055210ac8" (UID: "887a5a95-46ea-4df2-9686-b56055210ac8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:45.146129 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.145953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "887a5a95-46ea-4df2-9686-b56055210ac8" (UID: "887a5a95-46ea-4df2-9686-b56055210ac8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.159096 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.159077    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:21:45.160038 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.160017    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "887a5a95-46ea-4df2-9686-b56055210ac8" (UID: "887a5a95-46ea-4df2-9686-b56055210ac8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.235514 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/957cd07b-8af6-4187-bb39-0a48d954d3e9-cabundle-cert\") pod \"957cd07b-8af6-4187-bb39-0a48d954d3e9\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") "
Sep 17 18:21:45.235621 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235544    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-model-cache\") pod \"957cd07b-8af6-4187-bb39-0a48d954d3e9\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") "
Sep 17 18:21:45.235621 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957cd07b-8af6-4187-bb39-0a48d954d3e9-tls-certs\") pod \"957cd07b-8af6-4187-bb39-0a48d954d3e9\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") "
Sep 17 18:21:45.235722 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-kserve-provision-location\") pod \"957cd07b-8af6-4187-bb39-0a48d954d3e9\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") "
Sep 17 18:21:45.235761 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nshp5\" (UniqueName: \"kubernetes.io/projected/957cd07b-8af6-4187-bb39-0a48d954d3e9-kube-api-access-nshp5\") pod \"957cd07b-8af6-4187-bb39-0a48d954d3e9\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") "
Sep 17 18:21:45.235803 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "957cd07b-8af6-4187-bb39-0a48d954d3e9" (UID: "957cd07b-8af6-4187-bb39-0a48d954d3e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.235803 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235786    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-dshm\") pod \"957cd07b-8af6-4187-bb39-0a48d954d3e9\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") "
Sep 17 18:21:45.235909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235810    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-home\") pod \"957cd07b-8af6-4187-bb39-0a48d954d3e9\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") "
Sep 17 18:21:45.235909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235843    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-tmp-dir\") pod \"957cd07b-8af6-4187-bb39-0a48d954d3e9\" (UID: \"957cd07b-8af6-4187-bb39-0a48d954d3e9\") "
Sep 17 18:21:45.235909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.235874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957cd07b-8af6-4187-bb39-0a48d954d3e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "957cd07b-8af6-4187-bb39-0a48d954d3e9" (UID: "957cd07b-8af6-4187-bb39-0a48d954d3e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:45.236166 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236148    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.236256 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236171    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.236256 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236184    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.236256 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236199    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/957cd07b-8af6-4187-bb39-0a48d954d3e9-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.236256 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236212    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887a5a95-46ea-4df2-9686-b56055210ac8-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.236256 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236249    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rc22\" (UniqueName: \"kubernetes.io/projected/887a5a95-46ea-4df2-9686-b56055210ac8-kube-api-access-5rc22\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.236514 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236263    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.236514 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236276    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.236514 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236288    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887a5a95-46ea-4df2-9686-b56055210ac8-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.236650 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.236519    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-home" (OuterVolumeSpecName: "home") pod "957cd07b-8af6-4187-bb39-0a48d954d3e9" (UID: "957cd07b-8af6-4187-bb39-0a48d954d3e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.237610 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.237583    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957cd07b-8af6-4187-bb39-0a48d954d3e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "957cd07b-8af6-4187-bb39-0a48d954d3e9" (UID: "957cd07b-8af6-4187-bb39-0a48d954d3e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:45.238033 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.238011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-dshm" (OuterVolumeSpecName: "dshm") pod "957cd07b-8af6-4187-bb39-0a48d954d3e9" (UID: "957cd07b-8af6-4187-bb39-0a48d954d3e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.238179 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.238159    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957cd07b-8af6-4187-bb39-0a48d954d3e9-kube-api-access-nshp5" (OuterVolumeSpecName: "kube-api-access-nshp5") pod "957cd07b-8af6-4187-bb39-0a48d954d3e9" (UID: "957cd07b-8af6-4187-bb39-0a48d954d3e9"). InnerVolumeSpecName "kube-api-access-nshp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:45.249697 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.249676    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "957cd07b-8af6-4187-bb39-0a48d954d3e9" (UID: "957cd07b-8af6-4187-bb39-0a48d954d3e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.257493 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.257472    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "957cd07b-8af6-4187-bb39-0a48d954d3e9" (UID: "957cd07b-8af6-4187-bb39-0a48d954d3e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:45.337528 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.337472    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nshp5\" (UniqueName: \"kubernetes.io/projected/957cd07b-8af6-4187-bb39-0a48d954d3e9-kube-api-access-nshp5\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.337528 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.337494    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.337528 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.337503    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.337528 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.337511    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.337528 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.337520    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/957cd07b-8af6-4187-bb39-0a48d954d3e9-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.337528 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.337529    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/957cd07b-8af6-4187-bb39-0a48d954d3e9-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:21:45.585674 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.585634    2581 generic.go:356] "Generic (PLEG): container finished" podID="887a5a95-46ea-4df2-9686-b56055210ac8" containerID="ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773" exitCode=0
Sep 17 18:21:45.585815 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.585678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" event={"ID":"887a5a95-46ea-4df2-9686-b56055210ac8","Type":"ContainerDied","Data":"ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773"}
Sep 17 18:21:45.585815 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.585776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2" event={"ID":"887a5a95-46ea-4df2-9686-b56055210ac8","Type":"ContainerDied","Data":"4f2f4c896d3ae6beb8d1da2f51d6d62e7dc2404b5f71b23c75ffef48122b747b"}
Sep 17 18:21:45.585815 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.585795    2581 scope.go:117] "RemoveContainer" containerID="ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773"
Sep 17 18:21:45.585815 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.585703    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"
Sep 17 18:21:45.588601 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.588536    2581 generic.go:356] "Generic (PLEG): container finished" podID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerID="e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784" exitCode=0
Sep 17 18:21:45.588601 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.588561    2581 generic.go:356] "Generic (PLEG): container finished" podID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerID="cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25" exitCode=0
Sep 17 18:21:45.588773 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.588627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" event={"ID":"957cd07b-8af6-4187-bb39-0a48d954d3e9","Type":"ContainerDied","Data":"e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784"}
Sep 17 18:21:45.588773 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.588698    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" event={"ID":"957cd07b-8af6-4187-bb39-0a48d954d3e9","Type":"ContainerDied","Data":"cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25"}
Sep 17 18:21:45.588773 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.588716    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"
Sep 17 18:21:45.588773 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.588728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h" event={"ID":"957cd07b-8af6-4187-bb39-0a48d954d3e9","Type":"ContainerDied","Data":"d32901bdeddad2018ecd8792b53f0d115a46d64f67ebc409ccf89c811e726321"}
Sep 17 18:21:45.596727 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.596581    2581 scope.go:117] "RemoveContainer" containerID="9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445"
Sep 17 18:21:45.605492 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.605472    2581 scope.go:117] "RemoveContainer" containerID="ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773"
Sep 17 18:21:45.605763 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:21:45.605746    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773\": container with ID starting with ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773 not found: ID does not exist" containerID="ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773"
Sep 17 18:21:45.605809 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.605769    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773"} err="failed to get container status \"ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773\": rpc error: code = NotFound desc = could not find container \"ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773\": container with ID starting with ce99e10444ce2804724cb34dab512deff8aeb553c6cd732cdab22f9e2c571773 not found: ID does not exist"
Sep 17 18:21:45.605809 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.605784    2581 scope.go:117] "RemoveContainer" containerID="9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445"
Sep 17 18:21:45.606102 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:21:45.606082    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445\": container with ID starting with 9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445 not found: ID does not exist" containerID="9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445"
Sep 17 18:21:45.606164 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.606107    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445"} err="failed to get container status \"9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445\": rpc error: code = NotFound desc = could not find container \"9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445\": container with ID starting with 9359bdb6bade2842b2deebc86af8cc7d36b7eff16e50d8a546d4d37695b00445 not found: ID does not exist"
Sep 17 18:21:45.606164 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.606121    2581 scope.go:117] "RemoveContainer" containerID="e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784"
Sep 17 18:21:45.610674 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.610646    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"]
Sep 17 18:21:45.615520 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.615497    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86m2mm2"]
Sep 17 18:21:45.616031 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.616004    2581 scope.go:117] "RemoveContainer" containerID="fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9"
Sep 17 18:21:45.624058 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.624039    2581 scope.go:117] "RemoveContainer" containerID="cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25"
Sep 17 18:21:45.628088 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.628066    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"]
Sep 17 18:21:45.632624 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.632598    2581 scope.go:117] "RemoveContainer" containerID="e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784"
Sep 17 18:21:45.632882 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:21:45.632863    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784\": container with ID starting with e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784 not found: ID does not exist" containerID="e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784"
Sep 17 18:21:45.633055 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.632890    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784"} err="failed to get container status \"e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784\": rpc error: code = NotFound desc = could not find container \"e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784\": container with ID starting with e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784 not found: ID does not exist"
Sep 17 18:21:45.633055 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.632906    2581 scope.go:117] "RemoveContainer" containerID="fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9"
Sep 17 18:21:45.633249 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:21:45.633227    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9\": container with ID starting with fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9 not found: ID does not exist" containerID="fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9"
Sep 17 18:21:45.633289 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.633255    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9"} err="failed to get container status \"fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9\": rpc error: code = NotFound desc = could not find container \"fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9\": container with ID starting with fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9 not found: ID does not exist"
Sep 17 18:21:45.633289 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.633276    2581 scope.go:117] "RemoveContainer" containerID="cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25"
Sep 17 18:21:45.633546 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:21:45.633525    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25\": container with ID starting with cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25 not found: ID does not exist" containerID="cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25"
Sep 17 18:21:45.633612 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.633549    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25"} err="failed to get container status \"cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25\": rpc error: code = NotFound desc = could not find container \"cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25\": container with ID starting with cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25 not found: ID does not exist"
Sep 17 18:21:45.633612 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.633563    2581 scope.go:117] "RemoveContainer" containerID="e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784"
Sep 17 18:21:45.633834 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.633808    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784"} err="failed to get container status \"e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784\": rpc error: code = NotFound desc = could not find container \"e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784\": container with ID starting with e6d0b914962d4464dace8b89976f31666dc337f16848ae22c161c4b461e48784 not found: ID does not exist"
Sep 17 18:21:45.633881 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.633837    2581 scope.go:117] "RemoveContainer" containerID="fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9"
Sep 17 18:21:45.634143 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.634118    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9"} err="failed to get container status \"fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9\": rpc error: code = NotFound desc = could not find container \"fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9\": container with ID starting with fcac54672691951a620798fe76721ed901b3fa42ab4b5a1e4433fadc5c4da0e9 not found: ID does not exist"
Sep 17 18:21:45.634217 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.634143    2581 scope.go:117] "RemoveContainer" containerID="cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25"
Sep 17 18:21:45.634429 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.634410    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25"} err="failed to get container status \"cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25\": rpc error: code = NotFound desc = could not find container \"cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25\": container with ID starting with cafbfaa1d49c90b9c86f381c873257ed91d0ea1dc0aa4eefdacdd543922aff25 not found: ID does not exist"
Sep 17 18:21:45.635137 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:45.635118    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5c78d6666bxg24h"]
Sep 17 18:21:47.181188 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:47.181153    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" path="/var/lib/kubelet/pods/887a5a95-46ea-4df2-9686-b56055210ac8/volumes"
Sep 17 18:21:47.181657 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:21:47.181641    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" path="/var/lib/kubelet/pods/957cd07b-8af6-4187-bb39-0a48d954d3e9/volumes"
Sep 17 18:22:18.186529 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:18.186495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:18.282891 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:18.282856    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:19.214955 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:19.214923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:21.103943 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:21.103900    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:23.386285 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386251    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"]
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386615    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="storage-initializer"
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386631    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="storage-initializer"
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386650    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main"
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386657    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main"
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386665    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="llm-d-routing-sidecar"
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386671    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="llm-d-routing-sidecar"
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386685    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main"
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386690    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main"
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386701    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="storage-initializer"
Sep 17 18:22:23.386730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386710    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="storage-initializer"
Sep 17 18:22:23.387102 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386777    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="887a5a95-46ea-4df2-9686-b56055210ac8" containerName="main"
Sep 17 18:22:23.387102 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386788    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="llm-d-routing-sidecar"
Sep 17 18:22:23.387102 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.386798    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="957cd07b-8af6-4187-bb39-0a48d954d3e9" containerName="main"
Sep 17 18:22:23.391534 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.391512    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.394795 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.394770    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 18:22:23.395001 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.394771    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 18:22:23.395094 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.394802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-mgmgd\""
Sep 17 18:22:23.395164 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.394922    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 18:22:23.395381 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.395362    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zpnsk\""
Sep 17 18:22:23.395476 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.395434    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 18:22:23.395538 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.395364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:22:23.405276 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.405251    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"]
Sep 17 18:22:23.416854 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.416803    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"]
Sep 17 18:22:23.420135 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.420118    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.435840 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.435816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"]
Sep 17 18:22:23.457334 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.457315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:23.520788 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.520768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6898105a-33df-4db6-9703-271087f57a8f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.520877 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.520797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.520877 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.520821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6898105a-33df-4db6-9703-271087f57a8f-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.520951 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.520909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-model-cache\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.520951 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.520932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820069c0-f923-41b6-82de-c49e311cb079-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.521055 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.520951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-dshm\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.521055 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.521055 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.521157 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xrk2\" (UniqueName: \"kubernetes.io/projected/6898105a-33df-4db6-9703-271087f57a8f-kube-api-access-5xrk2\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.521157 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.521157 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.521252 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk2cd\" (UniqueName: \"kubernetes.io/projected/820069c0-f923-41b6-82de-c49e311cb079-kube-api-access-nk2cd\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.521252 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820069c0-f923-41b6-82de-c49e311cb079-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.521252 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.521354 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.521354 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.521277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-home\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.621814 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.621790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.621911 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.621819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-home\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.621911 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.621848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6898105a-33df-4db6-9703-271087f57a8f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.622030 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.621992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.622030 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6898105a-33df-4db6-9703-271087f57a8f-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.622139 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-model-cache\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.622139 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820069c0-f923-41b6-82de-c49e311cb079-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.622139 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-dshm\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.622303 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.622303 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.622303 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xrk2\" (UniqueName: \"kubernetes.io/projected/6898105a-33df-4db6-9703-271087f57a8f-kube-api-access-5xrk2\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.622303 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.622303 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.622303 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.622627 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk2cd\" (UniqueName: \"kubernetes.io/projected/820069c0-f923-41b6-82de-c49e311cb079-kube-api-access-nk2cd\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.622627 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820069c0-f923-41b6-82de-c49e311cb079-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.622627 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.622627 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.622627 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-model-cache\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.622916 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-home\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.622916 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.622808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6898105a-33df-4db6-9703-271087f57a8f-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.623079 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.623052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.623260 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.623235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.623334 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.623301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.623334 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.623305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.623454 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.623418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820069c0-f923-41b6-82de-c49e311cb079-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.624646 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.624612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.624757 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.624653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6898105a-33df-4db6-9703-271087f57a8f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.625029 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.625011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820069c0-f923-41b6-82de-c49e311cb079-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.625276 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.625259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-dshm\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.640051 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.639940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk2cd\" (UniqueName: \"kubernetes.io/projected/820069c0-f923-41b6-82de-c49e311cb079-kube-api-access-nk2cd\") pod \"custom-route-timeout-pd-test-kserve-694cb954ff-cmv59\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.640154 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.640084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xrk2\" (UniqueName: \"kubernetes.io/projected/6898105a-33df-4db6-9703-271087f57a8f-kube-api-access-5xrk2\") pod \"custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.701617 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.701595    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:23.703631 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.703603    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:23.731489 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.731470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:23.983739 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:23.983712    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"]
Sep 17 18:22:23.984635 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:22:23.984612    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820069c0_f923_41b6_82de_c49e311cb079.slice/crio-5ef530eff5c555a2dc8a4220c603953d75ef301023cc4fcb3ebb89d39f730991 WatchSource:0}: Error finding container 5ef530eff5c555a2dc8a4220c603953d75ef301023cc4fcb3ebb89d39f730991: Status 404 returned error can't find the container with id 5ef530eff5c555a2dc8a4220c603953d75ef301023cc4fcb3ebb89d39f730991
Sep 17 18:22:24.015420 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:24.015395    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"]
Sep 17 18:22:24.016654 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:22:24.016634    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6898105a_33df_4db6_9703_271087f57a8f.slice/crio-5391c6c75a4687cba829fbe8b34acfbf0712754a85be5e7e4eb84127d1620c0c WatchSource:0}: Error finding container 5391c6c75a4687cba829fbe8b34acfbf0712754a85be5e7e4eb84127d1620c0c: Status 404 returned error can't find the container with id 5391c6c75a4687cba829fbe8b34acfbf0712754a85be5e7e4eb84127d1620c0c
Sep 17 18:22:24.812455 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:24.812420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" event={"ID":"6898105a-33df-4db6-9703-271087f57a8f","Type":"ContainerStarted","Data":"e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262"}
Sep 17 18:22:24.812956 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:24.812470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" event={"ID":"6898105a-33df-4db6-9703-271087f57a8f","Type":"ContainerStarted","Data":"5391c6c75a4687cba829fbe8b34acfbf0712754a85be5e7e4eb84127d1620c0c"}
Sep 17 18:22:24.814168 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:24.814139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" event={"ID":"820069c0-f923-41b6-82de-c49e311cb079","Type":"ContainerStarted","Data":"a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a"}
Sep 17 18:22:24.814284 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:24.814187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" event={"ID":"820069c0-f923-41b6-82de-c49e311cb079","Type":"ContainerStarted","Data":"5ef530eff5c555a2dc8a4220c603953d75ef301023cc4fcb3ebb89d39f730991"}
Sep 17 18:22:24.814284 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:24.814211    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:25.825485 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:25.825437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" event={"ID":"820069c0-f923-41b6-82de-c49e311cb079","Type":"ContainerStarted","Data":"bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234"}
Sep 17 18:22:26.834436 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:26.834397    2581 generic.go:356] "Generic (PLEG): container finished" podID="6898105a-33df-4db6-9703-271087f57a8f" containerID="e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262" exitCode=0
Sep 17 18:22:26.834845 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:26.834458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" event={"ID":"6898105a-33df-4db6-9703-271087f57a8f","Type":"ContainerDied","Data":"e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262"}
Sep 17 18:22:27.364879 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:27.364850    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:27.841738 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:27.841705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" event={"ID":"6898105a-33df-4db6-9703-271087f57a8f","Type":"ContainerStarted","Data":"1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334"}
Sep 17 18:22:27.843801 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:27.843777    2581 generic.go:356] "Generic (PLEG): container finished" podID="820069c0-f923-41b6-82de-c49e311cb079" containerID="bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234" exitCode=0
Sep 17 18:22:27.843910 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:27.843810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" event={"ID":"820069c0-f923-41b6-82de-c49e311cb079","Type":"ContainerDied","Data":"bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234"}
Sep 17 18:22:27.867766 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:27.867715    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podStartSLOduration=2.570277382 podStartE2EDuration="4.867696286s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.297418904s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:27.865733995 +0000 UTC m=+2910.790205978" watchObservedRunningTime="2026-09-17 18:22:27.867696286 +0000 UTC m=+2910.792168246"
Sep 17 18:22:28.852633 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:28.852583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" event={"ID":"820069c0-f923-41b6-82de-c49e311cb079","Type":"ContainerStarted","Data":"04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55"}
Sep 17 18:22:28.885418 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:28.885360    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podStartSLOduration=3.503626214 podStartE2EDuration="5.885340091s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.381713877s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:28.881614863 +0000 UTC m=+2911.806086835" watchObservedRunningTime="2026-09-17 18:22:28.885340091 +0000 UTC m=+2911.809812051"
Sep 17 18:22:30.689146 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:30.689115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:33.217468 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:33.217433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:33.316386 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:33.316353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:33.704065 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:33.704026    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:33.704232 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:33.704128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:33.705593 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:33.705566    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:22:33.732632 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:33.732603    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:33.732782 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:33.732669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:22:33.734260 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:33.734230    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:22:40.888317 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:40.888278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:43.704548 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:43.704494    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:22:43.717049 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:43.717016    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:22:43.732696 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:43.732656    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:22:53.704703 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:53.704657    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:22:53.732136 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:53.732102    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:22:55.776888 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:55.776853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:55.837227 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:22:55.837194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:03.704167 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:03.704124    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:23:03.732610 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:03.732573    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:23:13.704743 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:13.704606    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:23:13.732654 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:13.732615    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:23:23.704492 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:23.704443    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:23:23.732704 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:23.732666    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:23:33.704810 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:33.704764    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:23:33.732749 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:33.732702    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:23:43.704833 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:43.704779    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:23:43.732755 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:43.732726    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:23:45.908709 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:45.908674    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:46.008521 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:46.008487    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:46.114059 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:46.114026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:52.201093 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:52.201058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:53.704693 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:53.704642    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:23:53.732504 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:53.732474    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:23:57.870323 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:57.870288    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:57.933260 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:23:57.933227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:00.090732 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:00.090704    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:24:00.097498 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:00.097480    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:24:03.704311 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:03.704265    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:24:03.732264 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:03.732232    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:24:05.460629 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:05.460598    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:13.704431 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:13.704371    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:24:13.732361 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:13.732331    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:24:23.704942 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:23.704886    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:24:23.732735 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:23.732704    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:24:33.704262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:33.704214    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:24:33.732182 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:33.732149    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:24:43.704600 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:43.704511    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:24:43.732243 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:43.732206    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:24:53.705012 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:53.704937    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 17 18:24:53.732308 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:24:53.732271    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:25:03.714457 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:03.714413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:25:03.733704 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:03.733672    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:25:03.743641 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:03.743622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:25:03.751677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:03.751653    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:25:13.091995 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:13.091931    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"]
Sep 17 18:25:13.095790 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:13.095759    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"]
Sep 17 18:25:13.108572 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:13.108535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:13.208118 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:13.208087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:28.116736 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:28.116695    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main" containerID="cri-o://04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55" gracePeriod=45
Sep 17 18:25:28.120860 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:28.120820    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main" containerID="cri-o://1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334" gracePeriod=45
Sep 17 18:25:28.937929 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:28.937895    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="llm-d-routing-sidecar" containerID="cri-o://a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a" gracePeriod=45
Sep 17 18:25:29.070710 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.070686    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:25:29.158603 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.158534    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6898105a-33df-4db6-9703-271087f57a8f-cabundle-cert\") pod \"6898105a-33df-4db6-9703-271087f57a8f\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") "
Sep 17 18:25:29.158603 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.158575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-dshm\") pod \"6898105a-33df-4db6-9703-271087f57a8f\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") "
Sep 17 18:25:29.159114 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.158605    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-home\") pod \"6898105a-33df-4db6-9703-271087f57a8f\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") "
Sep 17 18:25:29.159114 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.158627    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6898105a-33df-4db6-9703-271087f57a8f-tls-certs\") pod \"6898105a-33df-4db6-9703-271087f57a8f\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") "
Sep 17 18:25:29.159114 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.158649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xrk2\" (UniqueName: \"kubernetes.io/projected/6898105a-33df-4db6-9703-271087f57a8f-kube-api-access-5xrk2\") pod \"6898105a-33df-4db6-9703-271087f57a8f\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") "
Sep 17 18:25:29.159114 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.158720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-kserve-provision-location\") pod \"6898105a-33df-4db6-9703-271087f57a8f\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") "
Sep 17 18:25:29.159114 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.158778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-model-cache\") pod \"6898105a-33df-4db6-9703-271087f57a8f\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") "
Sep 17 18:25:29.159114 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.158817    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-tmp-dir\") pod \"6898105a-33df-4db6-9703-271087f57a8f\" (UID: \"6898105a-33df-4db6-9703-271087f57a8f\") "
Sep 17 18:25:29.159114 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.158984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6898105a-33df-4db6-9703-271087f57a8f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6898105a-33df-4db6-9703-271087f57a8f" (UID: "6898105a-33df-4db6-9703-271087f57a8f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:29.159486 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.159132    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6898105a-33df-4db6-9703-271087f57a8f-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.159873 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.159847    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-home" (OuterVolumeSpecName: "home") pod "6898105a-33df-4db6-9703-271087f57a8f" (UID: "6898105a-33df-4db6-9703-271087f57a8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.160001 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.159934    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6898105a-33df-4db6-9703-271087f57a8f" (UID: "6898105a-33df-4db6-9703-271087f57a8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.162891 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.162868    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-dshm" (OuterVolumeSpecName: "dshm") pod "6898105a-33df-4db6-9703-271087f57a8f" (UID: "6898105a-33df-4db6-9703-271087f57a8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.163003 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.162957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6898105a-33df-4db6-9703-271087f57a8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6898105a-33df-4db6-9703-271087f57a8f" (UID: "6898105a-33df-4db6-9703-271087f57a8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:29.163063 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.163029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6898105a-33df-4db6-9703-271087f57a8f-kube-api-access-5xrk2" (OuterVolumeSpecName: "kube-api-access-5xrk2") pod "6898105a-33df-4db6-9703-271087f57a8f" (UID: "6898105a-33df-4db6-9703-271087f57a8f"). InnerVolumeSpecName "kube-api-access-5xrk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:29.170744 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.170721    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6898105a-33df-4db6-9703-271087f57a8f" (UID: "6898105a-33df-4db6-9703-271087f57a8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.183279 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.183249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6898105a-33df-4db6-9703-271087f57a8f" (UID: "6898105a-33df-4db6-9703-271087f57a8f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.188662 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.188644    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:25:29.259498 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.259474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-dshm\") pod \"820069c0-f923-41b6-82de-c49e311cb079\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") "
Sep 17 18:25:29.259596 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.259508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820069c0-f923-41b6-82de-c49e311cb079-cabundle-cert\") pod \"820069c0-f923-41b6-82de-c49e311cb079\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") "
Sep 17 18:25:29.259596 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.259552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-kserve-provision-location\") pod \"820069c0-f923-41b6-82de-c49e311cb079\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") "
Sep 17 18:25:29.259596 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.259576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk2cd\" (UniqueName: \"kubernetes.io/projected/820069c0-f923-41b6-82de-c49e311cb079-kube-api-access-nk2cd\") pod \"820069c0-f923-41b6-82de-c49e311cb079\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") "
Sep 17 18:25:29.259767 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.259603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820069c0-f923-41b6-82de-c49e311cb079-tls-certs\") pod \"820069c0-f923-41b6-82de-c49e311cb079\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") "
Sep 17 18:25:29.259767 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.259626    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-tmp-dir\") pod \"820069c0-f923-41b6-82de-c49e311cb079\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") "
Sep 17 18:25:29.259767 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.259676    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-model-cache\") pod \"820069c0-f923-41b6-82de-c49e311cb079\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") "
Sep 17 18:25:29.259767 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.259722    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-home\") pod \"820069c0-f923-41b6-82de-c49e311cb079\" (UID: \"820069c0-f923-41b6-82de-c49e311cb079\") "
Sep 17 18:25:29.260026 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.259961    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-model-cache" (OuterVolumeSpecName: "model-cache") pod "820069c0-f923-41b6-82de-c49e311cb079" (UID: "820069c0-f923-41b6-82de-c49e311cb079"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.260180 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260005    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820069c0-f923-41b6-82de-c49e311cb079-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "820069c0-f923-41b6-82de-c49e311cb079" (UID: "820069c0-f923-41b6-82de-c49e311cb079"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:29.260457 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260437    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.260544 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260460    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.260544 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260474    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6898105a-33df-4db6-9703-271087f57a8f-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.260544 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260487    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xrk2\" (UniqueName: \"kubernetes.io/projected/6898105a-33df-4db6-9703-271087f57a8f-kube-api-access-5xrk2\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.260544 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260500    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820069c0-f923-41b6-82de-c49e311cb079-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.260544 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260514    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.260544 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260526    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.260544 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260539    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.260865 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260551    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6898105a-33df-4db6-9703-271087f57a8f-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.260865 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.260601    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-home" (OuterVolumeSpecName: "home") pod "820069c0-f923-41b6-82de-c49e311cb079" (UID: "820069c0-f923-41b6-82de-c49e311cb079"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.261689 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.261667    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-dshm" (OuterVolumeSpecName: "dshm") pod "820069c0-f923-41b6-82de-c49e311cb079" (UID: "820069c0-f923-41b6-82de-c49e311cb079"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.262459 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.262434    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820069c0-f923-41b6-82de-c49e311cb079-kube-api-access-nk2cd" (OuterVolumeSpecName: "kube-api-access-nk2cd") pod "820069c0-f923-41b6-82de-c49e311cb079" (UID: "820069c0-f923-41b6-82de-c49e311cb079"). InnerVolumeSpecName "kube-api-access-nk2cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:29.262607 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.262589    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/820069c0-f923-41b6-82de-c49e311cb079-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "820069c0-f923-41b6-82de-c49e311cb079" (UID: "820069c0-f923-41b6-82de-c49e311cb079"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:29.279016 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.278993    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "820069c0-f923-41b6-82de-c49e311cb079" (UID: "820069c0-f923-41b6-82de-c49e311cb079"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.288692 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.288672    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "820069c0-f923-41b6-82de-c49e311cb079" (UID: "820069c0-f923-41b6-82de-c49e311cb079"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:29.361668 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.361649    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.361668 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.361667    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.361786 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.361676    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.361786 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.361687    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk2cd\" (UniqueName: \"kubernetes.io/projected/820069c0-f923-41b6-82de-c49e311cb079-kube-api-access-nk2cd\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.361786 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.361695    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820069c0-f923-41b6-82de-c49e311cb079-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.361786 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.361702    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820069c0-f923-41b6-82de-c49e311cb079-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:25:29.898199 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.898167    2581 generic.go:356] "Generic (PLEG): container finished" podID="6898105a-33df-4db6-9703-271087f57a8f" containerID="1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334" exitCode=0
Sep 17 18:25:29.898322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.898211    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"
Sep 17 18:25:29.898322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.898249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" event={"ID":"6898105a-33df-4db6-9703-271087f57a8f","Type":"ContainerDied","Data":"1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334"}
Sep 17 18:25:29.898322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.898315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc" event={"ID":"6898105a-33df-4db6-9703-271087f57a8f","Type":"ContainerDied","Data":"5391c6c75a4687cba829fbe8b34acfbf0712754a85be5e7e4eb84127d1620c0c"}
Sep 17 18:25:29.898493 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.898352    2581 scope.go:117] "RemoveContainer" containerID="1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334"
Sep 17 18:25:29.900332 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.900308    2581 generic.go:356] "Generic (PLEG): container finished" podID="820069c0-f923-41b6-82de-c49e311cb079" containerID="04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55" exitCode=0
Sep 17 18:25:29.900332 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.900332    2581 generic.go:356] "Generic (PLEG): container finished" podID="820069c0-f923-41b6-82de-c49e311cb079" containerID="a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a" exitCode=0
Sep 17 18:25:29.900518 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.900335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" event={"ID":"820069c0-f923-41b6-82de-c49e311cb079","Type":"ContainerDied","Data":"04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55"}
Sep 17 18:25:29.900518 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.900383    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"
Sep 17 18:25:29.900518 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.900386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" event={"ID":"820069c0-f923-41b6-82de-c49e311cb079","Type":"ContainerDied","Data":"a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a"}
Sep 17 18:25:29.900518 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.900415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59" event={"ID":"820069c0-f923-41b6-82de-c49e311cb079","Type":"ContainerDied","Data":"5ef530eff5c555a2dc8a4220c603953d75ef301023cc4fcb3ebb89d39f730991"}
Sep 17 18:25:29.908871 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.908849    2581 scope.go:117] "RemoveContainer" containerID="e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262"
Sep 17 18:25:29.917082 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.917062    2581 scope.go:117] "RemoveContainer" containerID="1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334"
Sep 17 18:25:29.917344 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:25:29.917326    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334\": container with ID starting with 1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334 not found: ID does not exist" containerID="1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334"
Sep 17 18:25:29.917396 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.917352    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334"} err="failed to get container status \"1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334\": rpc error: code = NotFound desc = could not find container \"1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334\": container with ID starting with 1297c4cd939dc27f99a913bb5c463597fa798155198e7fefdd086040b323c334 not found: ID does not exist"
Sep 17 18:25:29.917396 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.917368    2581 scope.go:117] "RemoveContainer" containerID="e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262"
Sep 17 18:25:29.917599 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:25:29.917582    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262\": container with ID starting with e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262 not found: ID does not exist" containerID="e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262"
Sep 17 18:25:29.917641 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.917602    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262"} err="failed to get container status \"e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262\": rpc error: code = NotFound desc = could not find container \"e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262\": container with ID starting with e2dfccfda0780adef3ebed5d85be6c67520f3ed6caae746b7f88d55e68967262 not found: ID does not exist"
Sep 17 18:25:29.917641 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.917614    2581 scope.go:117] "RemoveContainer" containerID="04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55"
Sep 17 18:25:29.923715 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.923692    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"]
Sep 17 18:25:29.928345 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.928319    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-78cb6587bf-v76cc"]
Sep 17 18:25:29.928896 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.928874    2581 scope.go:117] "RemoveContainer" containerID="bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234"
Sep 17 18:25:29.937757 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.937736    2581 scope.go:117] "RemoveContainer" containerID="a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a"
Sep 17 18:25:29.942197 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.941713    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"]
Sep 17 18:25:29.948563 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.948535    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694cb954ff-cmv59"]
Sep 17 18:25:29.949547 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.949526    2581 scope.go:117] "RemoveContainer" containerID="04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55"
Sep 17 18:25:29.949834 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:25:29.949813    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55\": container with ID starting with 04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55 not found: ID does not exist" containerID="04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55"
Sep 17 18:25:29.949901 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.949840    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55"} err="failed to get container status \"04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55\": rpc error: code = NotFound desc = could not find container \"04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55\": container with ID starting with 04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55 not found: ID does not exist"
Sep 17 18:25:29.949901 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.949858    2581 scope.go:117] "RemoveContainer" containerID="bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234"
Sep 17 18:25:29.950154 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:25:29.950137    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234\": container with ID starting with bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234 not found: ID does not exist" containerID="bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234"
Sep 17 18:25:29.950205 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.950158    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234"} err="failed to get container status \"bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234\": rpc error: code = NotFound desc = could not find container \"bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234\": container with ID starting with bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234 not found: ID does not exist"
Sep 17 18:25:29.950205 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.950172    2581 scope.go:117] "RemoveContainer" containerID="a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a"
Sep 17 18:25:29.950470 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:25:29.950448    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a\": container with ID starting with a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a not found: ID does not exist" containerID="a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a"
Sep 17 18:25:29.950521 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.950476    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a"} err="failed to get container status \"a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a\": rpc error: code = NotFound desc = could not find container \"a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a\": container with ID starting with a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a not found: ID does not exist"
Sep 17 18:25:29.950521 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.950497    2581 scope.go:117] "RemoveContainer" containerID="04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55"
Sep 17 18:25:29.950760 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.950740    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55"} err="failed to get container status \"04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55\": rpc error: code = NotFound desc = could not find container \"04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55\": container with ID starting with 04e93ebee006f93f524a441906290a644d89f3a2babc3df102bd62bb25c39b55 not found: ID does not exist"
Sep 17 18:25:29.950818 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.950760    2581 scope.go:117] "RemoveContainer" containerID="bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234"
Sep 17 18:25:29.951019 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.951000    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234"} err="failed to get container status \"bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234\": rpc error: code = NotFound desc = could not find container \"bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234\": container with ID starting with bea2dccbfe877ffd7eb68f91e2f6f4649ce268bc52cbc6e533f635dcf05b3234 not found: ID does not exist"
Sep 17 18:25:29.951068 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.951020    2581 scope.go:117] "RemoveContainer" containerID="a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a"
Sep 17 18:25:29.951244 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:29.951224    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a"} err="failed to get container status \"a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a\": rpc error: code = NotFound desc = could not find container \"a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a\": container with ID starting with a3aa8dab74c91817a3ea9e21d0522e39e46b663ffd93cdd5c3dc81582f22099a not found: ID does not exist"
Sep 17 18:25:31.181513 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:31.181483    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6898105a-33df-4db6-9703-271087f57a8f" path="/var/lib/kubelet/pods/6898105a-33df-4db6-9703-271087f57a8f/volumes"
Sep 17 18:25:31.181943 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:31.181929    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="820069c0-f923-41b6-82de-c49e311cb079" path="/var/lib/kubelet/pods/820069c0-f923-41b6-82de-c49e311cb079/volumes"
Sep 17 18:25:52.740522 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:52.740490    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:59.397819 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:59.397790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:59.450225 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:25:59.450194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:03.075339 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:03.075292    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:03.173570 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:03.173539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:03.358669 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:03.358586    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:03.489857 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:03.489826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:04.683357 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:04.683316    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:06.967440 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:06.967408    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:09.680172 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:09.680128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:13.154276 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:13.154184    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:13.252201 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:13.252171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:13.352546 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:13.352521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:15.808434 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:15.808396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:15.909154 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:15.909120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:18.208316 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208281    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"]
Sep 17 18:26:18.208782 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208762    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="storage-initializer"
Sep 17 18:26:18.208832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208786    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="storage-initializer"
Sep 17 18:26:18.208832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208806    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="llm-d-routing-sidecar"
Sep 17 18:26:18.208832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208814    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="llm-d-routing-sidecar"
Sep 17 18:26:18.208832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208825    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main"
Sep 17 18:26:18.208982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208836    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main"
Sep 17 18:26:18.208982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208869    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main"
Sep 17 18:26:18.208982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208878    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main"
Sep 17 18:26:18.208982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208890    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="storage-initializer"
Sep 17 18:26:18.208982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.208898    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="storage-initializer"
Sep 17 18:26:18.209135 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.209026    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6898105a-33df-4db6-9703-271087f57a8f" containerName="main"
Sep 17 18:26:18.209135 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.209041    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="llm-d-routing-sidecar"
Sep 17 18:26:18.209135 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.209051    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="820069c0-f923-41b6-82de-c49e311cb079" containerName="main"
Sep 17 18:26:18.214054 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.214031    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.216759 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.216726    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 18:26:18.217077 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.217056    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 18:26:18.217252 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.217219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 18:26:18.217379 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.217361    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:26:18.217590 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.217573    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-zbb9p\""
Sep 17 18:26:18.217855 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.217826    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 18:26:18.223910 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.223887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"]
Sep 17 18:26:18.242846 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.242818    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:18.334929 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.334905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-home\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.335059 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.334933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzv8\" (UniqueName: \"kubernetes.io/projected/d18ed327-500e-47ea-8f1e-9130f5e3145b-kube-api-access-sqzv8\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.335059 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.334959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.335059 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.335058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.335168 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.335111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d18ed327-500e-47ea-8f1e-9130f5e3145b-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.335168 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.335146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.335168 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.335164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.335315 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.335254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d18ed327-500e-47ea-8f1e-9130f5e3145b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.435916 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.435893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d18ed327-500e-47ea-8f1e-9130f5e3145b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436036 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.435928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-home\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436036 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.435945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzv8\" (UniqueName: \"kubernetes.io/projected/d18ed327-500e-47ea-8f1e-9130f5e3145b-kube-api-access-sqzv8\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436036 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.435994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436167 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.436110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436167 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.436156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d18ed327-500e-47ea-8f1e-9130f5e3145b-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436272 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.436197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436272 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.436225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436430 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.436407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-home\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436491 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.436437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436569 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.436549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436656 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.436640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.436715 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.436675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d18ed327-500e-47ea-8f1e-9130f5e3145b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.438444 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.438428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.438765 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.438748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d18ed327-500e-47ea-8f1e-9130f5e3145b-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.446570 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.446542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzv8\" (UniqueName: \"kubernetes.io/projected/d18ed327-500e-47ea-8f1e-9130f5e3145b-kube-api-access-sqzv8\") pod \"router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.506596 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.506529    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:18.527125 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.527098    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:18.608324 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.608297    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:18.808344 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:18.808315    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"]
Sep 17 18:26:18.808344 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:26:18.808336    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18ed327_500e_47ea_8f1e_9130f5e3145b.slice/crio-4f5c1bdfa43fb34c8ade3e725da0e8eb77d823eaca341f02211b8d9dcb9b5c14 WatchSource:0}: Error finding container 4f5c1bdfa43fb34c8ade3e725da0e8eb77d823eaca341f02211b8d9dcb9b5c14: Status 404 returned error can't find the container with id 4f5c1bdfa43fb34c8ade3e725da0e8eb77d823eaca341f02211b8d9dcb9b5c14
Sep 17 18:26:19.190312 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:19.190227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" event={"ID":"d18ed327-500e-47ea-8f1e-9130f5e3145b","Type":"ContainerStarted","Data":"20fcd60b3bb6e8c44ccd5862ca90cf3894aa285eb50b23fdf62d9bdfee5718f9"}
Sep 17 18:26:19.190312 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:19.190279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" event={"ID":"d18ed327-500e-47ea-8f1e-9130f5e3145b","Type":"ContainerStarted","Data":"4f5c1bdfa43fb34c8ade3e725da0e8eb77d823eaca341f02211b8d9dcb9b5c14"}
Sep 17 18:26:19.530465 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:19.530437    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:19.914255 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:19.914185    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:20.016054 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:20.016013    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:21.201266 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:21.201240    2581 generic.go:356] "Generic (PLEG): container finished" podID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerID="20fcd60b3bb6e8c44ccd5862ca90cf3894aa285eb50b23fdf62d9bdfee5718f9" exitCode=0
Sep 17 18:26:21.201574 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:21.201312    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" event={"ID":"d18ed327-500e-47ea-8f1e-9130f5e3145b","Type":"ContainerDied","Data":"20fcd60b3bb6e8c44ccd5862ca90cf3894aa285eb50b23fdf62d9bdfee5718f9"}
Sep 17 18:26:22.207849 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:22.207810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" event={"ID":"d18ed327-500e-47ea-8f1e-9130f5e3145b","Type":"ContainerStarted","Data":"7a879d529542d3c128659c4a47666b83d17b22806fc47c3e1a90ae7eaedbb919"}
Sep 17 18:26:22.229948 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:22.229892    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podStartSLOduration=2.155363726 podStartE2EDuration="4.229874297s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.074510571s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:22.22839797 +0000 UTC m=+3145.152869929" watchObservedRunningTime="2026-09-17 18:26:22.229874297 +0000 UTC m=+3145.154346255"
Sep 17 18:26:28.527207 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:28.527169    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:28.527699 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:28.527257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:26:28.528795 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:28.528768    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:26:38.527770 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:38.527717    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:26:48.528360 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:48.528320    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:26:58.527653 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:26:58.527604    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:27:08.528436 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:08.528395    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:27:16.205262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.205230    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"]
Sep 17 18:27:16.208227 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.208203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.210753 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.210724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:27:16.210879 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.210782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 18:27:16.210961 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.210898    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 18:27:16.211086 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.211065    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 18:27:16.211176 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.211124    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 18:27:16.211224 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.211125    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-9w75r\""
Sep 17 18:27:16.219242 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.219223    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"]
Sep 17 18:27:16.237155 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.237135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:16.311385 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.311356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-model-cache\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.311520 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.311393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8ws\" (UniqueName: \"kubernetes.io/projected/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kube-api-access-8g8ws\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.311520 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.311428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tls-certs\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.311520 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.311499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.311659 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.311571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/890b1f83-5cd4-430d-8cfe-96edf2576ea9-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.311700 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.311669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.311700 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.311692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-home\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.311779 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.311716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-dshm\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.412914 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.412879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-model-cache\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413102 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.412930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8ws\" (UniqueName: \"kubernetes.io/projected/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kube-api-access-8g8ws\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413102 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.412993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tls-certs\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413102 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413102 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/890b1f83-5cd4-430d-8cfe-96edf2576ea9-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413330 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413330 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-home\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413330 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-dshm\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413330 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-model-cache\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413541 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413596 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413646 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-home\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.413823 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.413807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/890b1f83-5cd4-430d-8cfe-96edf2576ea9-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.415472 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.415451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tls-certs\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.415733 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.415718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-dshm\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.421753 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.421729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8ws\" (UniqueName: \"kubernetes.io/projected/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kube-api-access-8g8ws\") pod \"lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.521930 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.521856    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:16.816511 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:16.815920    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"]
Sep 17 18:27:16.824726 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:27:16.823195    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890b1f83_5cd4_430d_8cfe_96edf2576ea9.slice/crio-6fb289b05a55d6f6516cd4af1f07a682d7413ede5e0c8ec0e9340d6870249edd WatchSource:0}: Error finding container 6fb289b05a55d6f6516cd4af1f07a682d7413ede5e0c8ec0e9340d6870249edd: Status 404 returned error can't find the container with id 6fb289b05a55d6f6516cd4af1f07a682d7413ede5e0c8ec0e9340d6870249edd
Sep 17 18:27:17.510712 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:17.510683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" event={"ID":"890b1f83-5cd4-430d-8cfe-96edf2576ea9","Type":"ContainerStarted","Data":"71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5"}
Sep 17 18:27:17.511115 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:17.511008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" event={"ID":"890b1f83-5cd4-430d-8cfe-96edf2576ea9","Type":"ContainerStarted","Data":"6fb289b05a55d6f6516cd4af1f07a682d7413ede5e0c8ec0e9340d6870249edd"}
Sep 17 18:27:18.529013 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:18.527913    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:27:20.537520 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:20.537482    2581 generic.go:356] "Generic (PLEG): container finished" podID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerID="71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5" exitCode=0
Sep 17 18:27:20.538025 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:20.537542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" event={"ID":"890b1f83-5cd4-430d-8cfe-96edf2576ea9","Type":"ContainerDied","Data":"71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5"}
Sep 17 18:27:21.558498 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:21.558467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" event={"ID":"890b1f83-5cd4-430d-8cfe-96edf2576ea9","Type":"ContainerStarted","Data":"f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f"}
Sep 17 18:27:21.584172 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:21.584099    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" podStartSLOduration=2.1392534850000002 podStartE2EDuration="5.58407932s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.444825835s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:21.580448788 +0000 UTC m=+3204.504920748" watchObservedRunningTime="2026-09-17 18:27:21.58407932 +0000 UTC m=+3204.508551276"
Sep 17 18:27:22.522648 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:22.522601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:26.522670 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:26.522632    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:27:26.524030 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:26.524004    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:27:28.528526 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:28.528473    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:27:36.522351 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:36.522300    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:27:38.528164 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:38.528120    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:27:46.522552 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:46.522462    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:27:48.528369 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:48.528323    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:27:56.523134 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:56.523090    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:27:58.528018 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:27:58.527951    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:28:06.522680 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:06.522637    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:28:08.528473 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:08.528419    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 18:28:16.522808 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:16.522762    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:28:18.538700 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:18.538666    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:28:18.546625 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:18.546596    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:28:26.523016 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:26.522946    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:28:28.147953 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:28.147877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:28.159056 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:28.159018    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"]
Sep 17 18:28:29.370136 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:29.370102    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:36.532588 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:36.532553    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:28:36.540597 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:36.540576    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:28:43.185447 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:43.185412    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main" containerID="cri-o://7a879d529542d3c128659c4a47666b83d17b22806fc47c3e1a90ae7eaedbb919" gracePeriod=45
Sep 17 18:28:44.073262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.073220    2581 generic.go:356] "Generic (PLEG): container finished" podID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerID="7a879d529542d3c128659c4a47666b83d17b22806fc47c3e1a90ae7eaedbb919" exitCode=0
Sep 17 18:28:44.073262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.073243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" event={"ID":"d18ed327-500e-47ea-8f1e-9130f5e3145b","Type":"ContainerDied","Data":"7a879d529542d3c128659c4a47666b83d17b22806fc47c3e1a90ae7eaedbb919"}
Sep 17 18:28:44.127361 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.127339    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:28:44.136177 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136148    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-home\") pod \"d18ed327-500e-47ea-8f1e-9130f5e3145b\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") "
Sep 17 18:28:44.136177 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136177    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-tmp-dir\") pod \"d18ed327-500e-47ea-8f1e-9130f5e3145b\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") "
Sep 17 18:28:44.136365 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136199    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqzv8\" (UniqueName: \"kubernetes.io/projected/d18ed327-500e-47ea-8f1e-9130f5e3145b-kube-api-access-sqzv8\") pod \"d18ed327-500e-47ea-8f1e-9130f5e3145b\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") "
Sep 17 18:28:44.136365 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136219    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-kserve-provision-location\") pod \"d18ed327-500e-47ea-8f1e-9130f5e3145b\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") "
Sep 17 18:28:44.136365 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136234    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-model-cache\") pod \"d18ed327-500e-47ea-8f1e-9130f5e3145b\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") "
Sep 17 18:28:44.136365 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136261    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d18ed327-500e-47ea-8f1e-9130f5e3145b-cabundle-cert\") pod \"d18ed327-500e-47ea-8f1e-9130f5e3145b\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") "
Sep 17 18:28:44.136365 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136318    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-dshm\") pod \"d18ed327-500e-47ea-8f1e-9130f5e3145b\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") "
Sep 17 18:28:44.136652 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136526    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d18ed327-500e-47ea-8f1e-9130f5e3145b" (UID: "d18ed327-500e-47ea-8f1e-9130f5e3145b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.136919 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136897    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-home" (OuterVolumeSpecName: "home") pod "d18ed327-500e-47ea-8f1e-9130f5e3145b" (UID: "d18ed327-500e-47ea-8f1e-9130f5e3145b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.136919 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.136891    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18ed327-500e-47ea-8f1e-9130f5e3145b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d18ed327-500e-47ea-8f1e-9130f5e3145b" (UID: "d18ed327-500e-47ea-8f1e-9130f5e3145b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:44.138921 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.138896    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18ed327-500e-47ea-8f1e-9130f5e3145b-kube-api-access-sqzv8" (OuterVolumeSpecName: "kube-api-access-sqzv8") pod "d18ed327-500e-47ea-8f1e-9130f5e3145b" (UID: "d18ed327-500e-47ea-8f1e-9130f5e3145b"). InnerVolumeSpecName "kube-api-access-sqzv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:44.139054 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.139014    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-dshm" (OuterVolumeSpecName: "dshm") pod "d18ed327-500e-47ea-8f1e-9130f5e3145b" (UID: "d18ed327-500e-47ea-8f1e-9130f5e3145b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.156688 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.156652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d18ed327-500e-47ea-8f1e-9130f5e3145b" (UID: "d18ed327-500e-47ea-8f1e-9130f5e3145b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.164834 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.164809    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d18ed327-500e-47ea-8f1e-9130f5e3145b" (UID: "d18ed327-500e-47ea-8f1e-9130f5e3145b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:44.237341 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.237271    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d18ed327-500e-47ea-8f1e-9130f5e3145b-tls-certs\") pod \"d18ed327-500e-47ea-8f1e-9130f5e3145b\" (UID: \"d18ed327-500e-47ea-8f1e-9130f5e3145b\") "
Sep 17 18:28:44.237730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.237489    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.237730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.237500    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.237730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.237510    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqzv8\" (UniqueName: \"kubernetes.io/projected/d18ed327-500e-47ea-8f1e-9130f5e3145b-kube-api-access-sqzv8\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.237730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.237519    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.237730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.237528    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.237730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.237535    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d18ed327-500e-47ea-8f1e-9130f5e3145b-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.237730 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.237542    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18ed327-500e-47ea-8f1e-9130f5e3145b-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:28:44.239334 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.239312    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d18ed327-500e-47ea-8f1e-9130f5e3145b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d18ed327-500e-47ea-8f1e-9130f5e3145b" (UID: "d18ed327-500e-47ea-8f1e-9130f5e3145b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:44.338024 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:44.337993    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d18ed327-500e-47ea-8f1e-9130f5e3145b-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:28:45.081877 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:45.081848    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"
Sep 17 18:28:45.082069 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:45.081843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r" event={"ID":"d18ed327-500e-47ea-8f1e-9130f5e3145b","Type":"ContainerDied","Data":"4f5c1bdfa43fb34c8ade3e725da0e8eb77d823eaca341f02211b8d9dcb9b5c14"}
Sep 17 18:28:45.082069 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:45.082013    2581 scope.go:117] "RemoveContainer" containerID="7a879d529542d3c128659c4a47666b83d17b22806fc47c3e1a90ae7eaedbb919"
Sep 17 18:28:45.092455 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:45.092438    2581 scope.go:117] "RemoveContainer" containerID="20fcd60b3bb6e8c44ccd5862ca90cf3894aa285eb50b23fdf62d9bdfee5718f9"
Sep 17 18:28:45.107596 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:45.107577    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"]
Sep 17 18:28:45.113130 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:45.113108    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-dc557bdc5-r2f4r"]
Sep 17 18:28:45.183846 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:28:45.183817    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" path="/var/lib/kubelet/pods/d18ed327-500e-47ea-8f1e-9130f5e3145b/volumes"
Sep 17 18:29:00.145335 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:00.145303    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:29:00.167843 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:00.167804    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:29:04.467891 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:04.467854    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n_890b1f83-5cd4-430d-8cfe-96edf2576ea9/storage-initializer/0.log"
Sep 17 18:29:04.477778 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:04.477753    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n_890b1f83-5cd4-430d-8cfe-96edf2576ea9/main/0.log"
Sep 17 18:29:05.549632 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:05.549593    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"]
Sep 17 18:29:06.256360 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:06.256304    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:18.284175 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:18.284095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:18.383322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:18.383276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:20.571679 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:20.571643    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main" containerID="cri-o://f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f" gracePeriod=45
Sep 17 18:29:21.224148 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.224121    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:29:21.281078 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.280954    2581 generic.go:356] "Generic (PLEG): container finished" podID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerID="f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f" exitCode=0
Sep 17 18:29:21.281078 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.281027    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"
Sep 17 18:29:21.281078 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.281039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" event={"ID":"890b1f83-5cd4-430d-8cfe-96edf2576ea9","Type":"ContainerDied","Data":"f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f"}
Sep 17 18:29:21.281318 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.281113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n" event={"ID":"890b1f83-5cd4-430d-8cfe-96edf2576ea9","Type":"ContainerDied","Data":"6fb289b05a55d6f6516cd4af1f07a682d7413ede5e0c8ec0e9340d6870249edd"}
Sep 17 18:29:21.281318 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.281153    2581 scope.go:117] "RemoveContainer" containerID="f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f"
Sep 17 18:29:21.290075 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.290055    2581 scope.go:117] "RemoveContainer" containerID="71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5"
Sep 17 18:29:21.298229 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.298212    2581 scope.go:117] "RemoveContainer" containerID="f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f"
Sep 17 18:29:21.298549 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:29:21.298526    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f\": container with ID starting with f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f not found: ID does not exist" containerID="f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f"
Sep 17 18:29:21.298644 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.298555    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f"} err="failed to get container status \"f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f\": rpc error: code = NotFound desc = could not find container \"f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f\": container with ID starting with f4c1ad5c5564753f7a6d3a2be084b9bb0faa7588bde36959b0b1cc557cb2782f not found: ID does not exist"
Sep 17 18:29:21.298644 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.298573    2581 scope.go:117] "RemoveContainer" containerID="71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5"
Sep 17 18:29:21.298824 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:29:21.298806    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5\": container with ID starting with 71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5 not found: ID does not exist" containerID="71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5"
Sep 17 18:29:21.298869 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.298829    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5"} err="failed to get container status \"71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5\": rpc error: code = NotFound desc = could not find container \"71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5\": container with ID starting with 71830b690631fa47bb928edc1f83d66c0a3a1abc6308a5a2f2516c46e98822f5 not found: ID does not exist"
Sep 17 18:29:21.344726 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.344688    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/890b1f83-5cd4-430d-8cfe-96edf2576ea9-cabundle-cert\") pod \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") "
Sep 17 18:29:21.344909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.344744    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g8ws\" (UniqueName: \"kubernetes.io/projected/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kube-api-access-8g8ws\") pod \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") "
Sep 17 18:29:21.344909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.344767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-home\") pod \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") "
Sep 17 18:29:21.344909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.344797    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tmp-dir\") pod \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") "
Sep 17 18:29:21.344909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.344832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tls-certs\") pod \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") "
Sep 17 18:29:21.344909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.344852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-dshm\") pod \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") "
Sep 17 18:29:21.344909 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.344885    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kserve-provision-location\") pod \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") "
Sep 17 18:29:21.345307 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.344958    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-model-cache\") pod \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\" (UID: \"890b1f83-5cd4-430d-8cfe-96edf2576ea9\") "
Sep 17 18:29:21.345307 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.345164    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890b1f83-5cd4-430d-8cfe-96edf2576ea9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "890b1f83-5cd4-430d-8cfe-96edf2576ea9" (UID: "890b1f83-5cd4-430d-8cfe-96edf2576ea9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:21.345307 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.345222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-home" (OuterVolumeSpecName: "home") pod "890b1f83-5cd4-430d-8cfe-96edf2576ea9" (UID: "890b1f83-5cd4-430d-8cfe-96edf2576ea9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:21.345529 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.345493    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-model-cache" (OuterVolumeSpecName: "model-cache") pod "890b1f83-5cd4-430d-8cfe-96edf2576ea9" (UID: "890b1f83-5cd4-430d-8cfe-96edf2576ea9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:21.347438 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.347413    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kube-api-access-8g8ws" (OuterVolumeSpecName: "kube-api-access-8g8ws") pod "890b1f83-5cd4-430d-8cfe-96edf2576ea9" (UID: "890b1f83-5cd4-430d-8cfe-96edf2576ea9"). InnerVolumeSpecName "kube-api-access-8g8ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:21.347632 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.347613    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "890b1f83-5cd4-430d-8cfe-96edf2576ea9" (UID: "890b1f83-5cd4-430d-8cfe-96edf2576ea9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:21.347753 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.347728    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-dshm" (OuterVolumeSpecName: "dshm") pod "890b1f83-5cd4-430d-8cfe-96edf2576ea9" (UID: "890b1f83-5cd4-430d-8cfe-96edf2576ea9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:21.358655 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.358625    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "890b1f83-5cd4-430d-8cfe-96edf2576ea9" (UID: "890b1f83-5cd4-430d-8cfe-96edf2576ea9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:21.367748 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.367722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "890b1f83-5cd4-430d-8cfe-96edf2576ea9" (UID: "890b1f83-5cd4-430d-8cfe-96edf2576ea9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:21.446649 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.446617    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.446649 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.446645    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.446649 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.446656    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890b1f83-5cd4-430d-8cfe-96edf2576ea9-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.446862 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.446663    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.446862 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.446673    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.446862 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.446683    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890b1f83-5cd4-430d-8cfe-96edf2576ea9-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.446862 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.446692    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/890b1f83-5cd4-430d-8cfe-96edf2576ea9-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.446862 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.446699    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g8ws\" (UniqueName: \"kubernetes.io/projected/890b1f83-5cd4-430d-8cfe-96edf2576ea9-kube-api-access-8g8ws\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.606067 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.606031    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"]
Sep 17 18:29:21.612335 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:21.612309    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bcb7cb485-r8w9n"]
Sep 17 18:29:23.180938 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:23.180903    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" path="/var/lib/kubelet/pods/890b1f83-5cd4-430d-8cfe-96edf2576ea9/volumes"
Sep 17 18:29:23.294200 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:23.294139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:26.114062 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:26.114013    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:26.213666 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:26.213636    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:26.315706 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:26.315674    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:26.415829 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:26.415756    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:26.513281 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:26.513254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:28.796395 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:28.796359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:28.891490 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:28.891405    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:29.485938 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:29.485907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:32.126148 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:32.126114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:32.202882 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:32.202840    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:32.424099 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:32.424011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:45.162950 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:45.162914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:45.797314 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:45.797282    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:51.510296 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:51.510254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:59.129655 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.129622    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"]
Sep 17 18:29:59.130230 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130208    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main"
Sep 17 18:29:59.130322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130233    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main"
Sep 17 18:29:59.130322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130251    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main"
Sep 17 18:29:59.130322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130259    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main"
Sep 17 18:29:59.130322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130269    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="storage-initializer"
Sep 17 18:29:59.130322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130277    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="storage-initializer"
Sep 17 18:29:59.130322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130314    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="storage-initializer"
Sep 17 18:29:59.130322 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130322    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="storage-initializer"
Sep 17 18:29:59.130700 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130419    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d18ed327-500e-47ea-8f1e-9130f5e3145b" containerName="main"
Sep 17 18:29:59.130700 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.130438    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="890b1f83-5cd4-430d-8cfe-96edf2576ea9" containerName="main"
Sep 17 18:29:59.132957 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.132936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.135832 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.135810    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 18:29:59.135948 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.135851    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:29:59.135948 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.135938    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 18:29:59.136129 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.136033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-52grw\""
Sep 17 18:29:59.136129 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.136041    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 18:29:59.136276 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.136261    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 18:29:59.140341 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.139879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-home\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.140341 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.139930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.140341 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.140033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.140341 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.140135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.140341 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.140193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.140341 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.140220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.140341 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.140243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.142756 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.142729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:59.144654 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.144633    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"]
Sep 17 18:29:59.241289 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241289 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241500 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-home\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241500 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241500 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241671 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241671 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241671 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwd7\" (UniqueName: \"kubernetes.io/projected/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kube-api-access-rfwd7\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241671 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241873 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241873 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.241989 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.241950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-home\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.242091 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.242073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.243766 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.243746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.244103 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.244087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.342822 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.342796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwd7\" (UniqueName: \"kubernetes.io/projected/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kube-api-access-rfwd7\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:29:59.351277 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:29:59.351254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwd7\" (UniqueName: \"kubernetes.io/projected/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kube-api-access-rfwd7\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:06.452398 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:06.452345    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.110555 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:09.110523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:09.117682 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:09.117656    2581 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:09.210860 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:09.210838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:09.210983 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:09.210918    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:09.214213 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:09.214192    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") DevicePath \"csi-97f08d230b4539e39b1a91a7b6e5c7c14a7be31760bbddf18853dead6cb6ee85\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:09.308788 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:09.308764    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/c640e3025f003c03c46dae8c816de9b130cbdc1424bf5ecca1cf2d33f4551660/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:09.329638 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:09.329618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:09.344275 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:09.344253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:09.622192 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:09.622167    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"]
Sep 17 18:30:09.623738 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:30:09.623710    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode253fbf5_2d98_4bad_b8b8_8cdd564f2ddd.slice/crio-fa501bc58cd073bde60a646a3bca0987367e06ee173cc64a14da007da7771835 WatchSource:0}: Error finding container fa501bc58cd073bde60a646a3bca0987367e06ee173cc64a14da007da7771835: Status 404 returned error can't find the container with id fa501bc58cd073bde60a646a3bca0987367e06ee173cc64a14da007da7771835
Sep 17 18:30:10.558626 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:10.558593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" event={"ID":"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd","Type":"ContainerStarted","Data":"4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4"}
Sep 17 18:30:10.559023 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:10.558642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" event={"ID":"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd","Type":"ContainerStarted","Data":"fa501bc58cd073bde60a646a3bca0987367e06ee173cc64a14da007da7771835"}
Sep 17 18:30:12.571381 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:12.571348    2581 generic.go:356] "Generic (PLEG): container finished" podID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerID="4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4" exitCode=0
Sep 17 18:30:12.571760 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:12.571422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" event={"ID":"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd","Type":"ContainerDied","Data":"4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4"}
Sep 17 18:30:13.579284 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:13.579240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" event={"ID":"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd","Type":"ContainerStarted","Data":"f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d"}
Sep 17 18:30:13.606787 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:13.606740    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podStartSLOduration=12.510217236 podStartE2EDuration="14.606727308s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.096510072s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:13.603956619 +0000 UTC m=+3376.528428577" watchObservedRunningTime="2026-09-17 18:30:13.606727308 +0000 UTC m=+3376.531199308"
Sep 17 18:30:14.345062 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:14.345025    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:14.978288 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:14.978257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.098673 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:15.098644    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.344521 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:19.344486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:30:19.346096 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:19.346070    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 18:30:29.344891 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:29.344849    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 18:30:39.344644 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:39.344589    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 18:30:49.345367 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:49.345273    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 18:30:52.950941 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:52.950908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.008385 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:53.008357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.681266 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:57.681232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:59.345316 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:30:59.345273    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 18:31:09.344789 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:09.344748    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 18:31:19.345349 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:19.345299    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 18:31:29.355110 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:29.355071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:31:29.363199 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:29.363169    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:31:37.705209 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:37.705178    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:42.907916 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:42.907880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:53.815981 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:53.815938    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x_e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd/storage-initializer/0.log"
Sep 17 18:31:53.825816 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:53.825785    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x_e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd/main/0.log"
Sep 17 18:31:54.865908 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:54.865872    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"]
Sep 17 18:31:54.873021 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:54.872993    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:57.736372 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:57.736336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:57.828702 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:31:57.828668    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:03.890211 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:03.890171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:06.452516 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.452480    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:06.597095 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.597069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:06.811024 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.810903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:06.884636 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.884607    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:32:06.887492 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.887476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:06.888356 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.888335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:06.890128 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.890109    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:32:06.890436 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.890415    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 18:32:06.890535 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.890451    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-p55qs\""
Sep 17 18:32:06.890535 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.890456    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 18:32:06.890654 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.890565    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-cwqqz\""
Sep 17 18:32:06.891533 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.891514    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 18:32:06.891644 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.891518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 18:32:06.902149 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.902124    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:32:06.995012 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.994980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:06.995119 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.995021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:06.995119 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.995066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/176154bb-04c7-4d42-9b08-ceec05c15e4c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:06.995119 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.995095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt22v\" (UniqueName: \"kubernetes.io/projected/176154bb-04c7-4d42-9b08-ceec05c15e4c-kube-api-access-wt22v\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:06.995227 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.995151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:06.995227 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.995179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:06.995227 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.995219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:06.995320 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:06.995254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176154bb-04c7-4d42-9b08-ceec05c15e4c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt22v\" (UniqueName: \"kubernetes.io/projected/176154bb-04c7-4d42-9b08-ceec05c15e4c-kube-api-access-wt22v\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096442 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096442 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176154bb-04c7-4d42-9b08-ceec05c15e4c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096557 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096557 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096557 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/176154bb-04c7-4d42-9b08-ceec05c15e4c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096706 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096770 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096858 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.096918 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.096838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.097141 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.097120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/176154bb-04c7-4d42-9b08-ceec05c15e4c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.098630 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.098601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.098826 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.098807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176154bb-04c7-4d42-9b08-ceec05c15e4c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.104996 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.104957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt22v\" (UniqueName: \"kubernetes.io/projected/176154bb-04c7-4d42-9b08-ceec05c15e4c-kube-api-access-wt22v\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.199007 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.198989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:32:07.474583 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:07.474560    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:32:07.476642 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:32:07.476616    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176154bb_04c7_4d42_9b08_ceec05c15e4c.slice/crio-897af56e9c3f0ab0ae5936195572aa2c3d175919ecfd4c1a49f491e5b9204895 WatchSource:0}: Error finding container 897af56e9c3f0ab0ae5936195572aa2c3d175919ecfd4c1a49f491e5b9204895: Status 404 returned error can't find the container with id 897af56e9c3f0ab0ae5936195572aa2c3d175919ecfd4c1a49f491e5b9204895
Sep 17 18:32:08.220120 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:08.220080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"176154bb-04c7-4d42-9b08-ceec05c15e4c","Type":"ContainerStarted","Data":"ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10"}
Sep 17 18:32:08.220314 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:08.220142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"176154bb-04c7-4d42-9b08-ceec05c15e4c","Type":"ContainerStarted","Data":"897af56e9c3f0ab0ae5936195572aa2c3d175919ecfd4c1a49f491e5b9204895"}
Sep 17 18:32:09.887101 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:09.887066    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main" containerID="cri-o://f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d" gracePeriod=45
Sep 17 18:32:09.916704 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:09.916678    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:10.017817 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.017728    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:10.233314 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.233282    2581 generic.go:356] "Generic (PLEG): container finished" podID="176154bb-04c7-4d42-9b08-ceec05c15e4c" containerID="ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10" exitCode=0
Sep 17 18:32:10.233489 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.233318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"176154bb-04c7-4d42-9b08-ceec05c15e4c","Type":"ContainerDied","Data":"ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10"}
Sep 17 18:32:10.356500 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.356470    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 18:32:10.537374 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.537348    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:32:10.625548 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-dshm\") pod \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") "
Sep 17 18:32:10.625702 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfwd7\" (UniqueName: \"kubernetes.io/projected/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kube-api-access-rfwd7\") pod \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") "
Sep 17 18:32:10.625702 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625596    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-cabundle-cert\") pod \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") "
Sep 17 18:32:10.625702 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-model-cache\") pod \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") "
Sep 17 18:32:10.625886 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") pod \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") "
Sep 17 18:32:10.625886 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kserve-provision-location\") pod \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") "
Sep 17 18:32:10.625886 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625825    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-home\") pod \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") "
Sep 17 18:32:10.626073 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tmp-dir\") pod \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") "
Sep 17 18:32:10.626073 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tls-certs\") pod \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\" (UID: \"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd\") "
Sep 17 18:32:10.626073 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-model-cache" (OuterVolumeSpecName: "model-cache") pod "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" (UID: "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:10.626073 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.625994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" (UID: "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:10.626278 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.626198    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:32:10.626278 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.626216    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:32:10.626278 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.626245    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-home" (OuterVolumeSpecName: "home") pod "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" (UID: "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:10.627935 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.627913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-dshm" (OuterVolumeSpecName: "dshm") pod "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" (UID: "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:10.628732 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.628677    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" (UID: "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:10.628851 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.628767    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kube-api-access-rfwd7" (OuterVolumeSpecName: "kube-api-access-rfwd7") pod "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" (UID: "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd"). InnerVolumeSpecName "kube-api-access-rfwd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:10.633001 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.632980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" (UID: "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd"). InnerVolumeSpecName "pvc-45be254d-0821-4e52-8cb1-7e118558d2b4". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:32:10.638039 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.638004    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" (UID: "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:10.650623 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.650600    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" (UID: "e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:10.727293 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.727265    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:32:10.727379 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.727299    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwd7\" (UniqueName: \"kubernetes.io/projected/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kube-api-access-rfwd7\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:32:10.727379 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.727342    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") on node \"ip-10-0-133-194.ec2.internal\" "
Sep 17 18:32:10.727379 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.727359    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:32:10.727379 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.727373    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:32:10.727501 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.727388    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:32:10.727501 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.727404    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:32:10.737591 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.737574    2581 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-45be254d-0821-4e52-8cb1-7e118558d2b4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf") on node "ip-10-0-133-194.ec2.internal"
Sep 17 18:32:10.828020 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:10.827946    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-45be254d-0821-4e52-8cb1-7e118558d2b4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f04744aa806acedf\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"csi-97f08d230b4539e39b1a91a7b6e5c7c14a7be31760bbddf18853dead6cb6ee85\""
Sep 17 18:32:11.240288 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.240246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"176154bb-04c7-4d42-9b08-ceec05c15e4c","Type":"ContainerStarted","Data":"6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78"}
Sep 17 18:32:11.241985 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.241940    2581 generic.go:356] "Generic (PLEG): container finished" podID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerID="f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d" exitCode=0
Sep 17 18:32:11.242094 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.241991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" event={"ID":"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd","Type":"ContainerDied","Data":"f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d"}
Sep 17 18:32:11.242094 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.242007    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"
Sep 17 18:32:11.242094 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.242048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x" event={"ID":"e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd","Type":"ContainerDied","Data":"fa501bc58cd073bde60a646a3bca0987367e06ee173cc64a14da007da7771835"}
Sep 17 18:32:11.242094 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.242079    2581 scope.go:117] "RemoveContainer" containerID="f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d"
Sep 17 18:32:11.250803 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.250782    2581 scope.go:117] "RemoveContainer" containerID="4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4"
Sep 17 18:32:11.258456 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.258427    2581 scope.go:117] "RemoveContainer" containerID="f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d"
Sep 17 18:32:11.258731 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:32:11.258713    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d\": container with ID starting with f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d not found: ID does not exist" containerID="f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d"
Sep 17 18:32:11.258795 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.258738    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d"} err="failed to get container status \"f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d\": rpc error: code = NotFound desc = could not find container \"f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d\": container with ID starting with f800092050b8f6a5d353f01566d4c7696575a74e67eb3488212cd417c7dfd44d not found: ID does not exist"
Sep 17 18:32:11.258795 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.258755    2581 scope.go:117] "RemoveContainer" containerID="4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4"
Sep 17 18:32:11.259013 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:32:11.258996    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4\": container with ID starting with 4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4 not found: ID does not exist" containerID="4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4"
Sep 17 18:32:11.259064 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.259018    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4"} err="failed to get container status \"4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4\": rpc error: code = NotFound desc = could not find container \"4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4\": container with ID starting with 4f631e2a4c44f27b34cf59491833aaecb81c09aa5d20c3a02987a4334803d0f4 not found: ID does not exist"
Sep 17 18:32:11.265836 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.265794    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.09064823 podStartE2EDuration="5.265782425s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.175134195s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:11.263562155 +0000 UTC m=+3494.188034113" watchObservedRunningTime="2026-09-17 18:32:11.265782425 +0000 UTC m=+3494.190254390"
Sep 17 18:32:11.282130 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.282106    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"]
Sep 17 18:32:11.287364 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:11.287341    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6d8987bf48-6658x"]
Sep 17 18:32:13.181841 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:13.181805    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" path="/var/lib/kubelet/pods/e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd/volumes"
Sep 17 18:32:15.422621 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:15.422586    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:15.522129 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:15.522097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:15.624740 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:15.624692    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:15.724646 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:15.724620    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:15.847332 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:15.847299    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:17.843897 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:17.843853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:17.896927 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:17.896886    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:18.308356 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.308329    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:18.681391 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.681314    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"]
Sep 17 18:32:18.681744 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.681725    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="storage-initializer"
Sep 17 18:32:18.681829 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.681746    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="storage-initializer"
Sep 17 18:32:18.681829 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.681761    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main"
Sep 17 18:32:18.681829 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.681769    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main"
Sep 17 18:32:18.681989 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.681885    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e253fbf5-2d98-4bad-b8b8-8cdd564f2ddd" containerName="main"
Sep 17 18:32:18.688613 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.688594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:18.691651 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.691626    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 18:32:18.691784 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.691686    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 18:32:18.691843 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.691629    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-pd27s\""
Sep 17 18:32:18.691904 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.691888    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 18:32:18.693091 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.693071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"]
Sep 17 18:32:18.797253 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.797225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbrm\" (UniqueName: \"kubernetes.io/projected/44270a4c-6bad-4213-9db0-de0812258e5a-kube-api-access-srbrm\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:18.817300 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.817273    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:18.897659 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.897636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbrm\" (UniqueName: \"kubernetes.io/projected/44270a4c-6bad-4213-9db0-de0812258e5a-kube-api-access-srbrm\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:18.906544 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:18.906525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbrm\" (UniqueName: \"kubernetes.io/projected/44270a4c-6bad-4213-9db0-de0812258e5a-kube-api-access-srbrm\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:20.747811 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:20.747776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:20.851901 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:20.851868    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:20.947631 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:20.947601    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:21.016170 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:21.016088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:21.816957 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:21.816923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:21.823919 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:21.823895    2581 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:21.917339 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:21.917316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:21.917459 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:21.917442    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:21.920673 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:21.920649    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") DevicePath \"csi-f7c53c27b3a4eb8f5b0f73b3272bfa9bdc7cabd18b407b8e56a8bb18204b254e\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:22.155957 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:22.155877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:22.295155 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:22.295118    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2b7e88d0f08fdc756cce5bfc50f5b573cede2d505463b07e3d2b15552285577b/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:22.310026 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:22.309996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"e2e-pvc-lora-download-sj7k8\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:22.599268 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:22.599224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:22.874334 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:22.874253    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"]
Sep 17 18:32:22.876941 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:32:22.876910    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44270a4c_6bad_4213_9db0_de0812258e5a.slice/crio-e6db616c6871d8356e6fe010e305e019cf83b346d3a80aae26342d3d55f71163 WatchSource:0}: Error finding container e6db616c6871d8356e6fe010e305e019cf83b346d3a80aae26342d3d55f71163: Status 404 returned error can't find the container with id e6db616c6871d8356e6fe010e305e019cf83b346d3a80aae26342d3d55f71163
Sep 17 18:32:23.310570 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:23.310532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8" event={"ID":"44270a4c-6bad-4213-9db0-de0812258e5a","Type":"ContainerStarted","Data":"b75e1b9f2b377491c589ef879e143a6ac357a632b29a42f3ea2a978231f2b3a5"}
Sep 17 18:32:23.310570 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:23.310582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8" event={"ID":"44270a4c-6bad-4213-9db0-de0812258e5a","Type":"ContainerStarted","Data":"e6db616c6871d8356e6fe010e305e019cf83b346d3a80aae26342d3d55f71163"}
Sep 17 18:32:23.329027 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:23.328960    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8" podStartSLOduration=8.328945802 podStartE2EDuration="8.328945802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:23.328138971 +0000 UTC m=+3506.252610929" watchObservedRunningTime="2026-09-17 18:32:23.328945802 +0000 UTC m=+3506.253417779"
Sep 17 18:32:25.323311 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:25.323279    2581 generic.go:356] "Generic (PLEG): container finished" podID="44270a4c-6bad-4213-9db0-de0812258e5a" containerID="b75e1b9f2b377491c589ef879e143a6ac357a632b29a42f3ea2a978231f2b3a5" exitCode=0
Sep 17 18:32:25.323761 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:25.323357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8" event={"ID":"44270a4c-6bad-4213-9db0-de0812258e5a","Type":"ContainerDied","Data":"b75e1b9f2b377491c589ef879e143a6ac357a632b29a42f3ea2a978231f2b3a5"}
Sep 17 18:32:26.464462 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:26.464435    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:26.558408 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:26.558379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srbrm\" (UniqueName: \"kubernetes.io/projected/44270a4c-6bad-4213-9db0-de0812258e5a-kube-api-access-srbrm\") pod \"44270a4c-6bad-4213-9db0-de0812258e5a\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") "
Sep 17 18:32:26.558534 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:26.558483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"44270a4c-6bad-4213-9db0-de0812258e5a\" (UID: \"44270a4c-6bad-4213-9db0-de0812258e5a\") "
Sep 17 18:32:26.560574 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:26.560540    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44270a4c-6bad-4213-9db0-de0812258e5a-kube-api-access-srbrm" (OuterVolumeSpecName: "kube-api-access-srbrm") pod "44270a4c-6bad-4213-9db0-de0812258e5a" (UID: "44270a4c-6bad-4213-9db0-de0812258e5a"). InnerVolumeSpecName "kube-api-access-srbrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:26.561233 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:26.561209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada" (OuterVolumeSpecName: "model-storage") pod "44270a4c-6bad-4213-9db0-de0812258e5a" (UID: "44270a4c-6bad-4213-9db0-de0812258e5a"). InnerVolumeSpecName "pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:32:26.659602 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:26.659535    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") on node \"ip-10-0-133-194.ec2.internal\" "
Sep 17 18:32:26.659602 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:26.659571    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srbrm\" (UniqueName: \"kubernetes.io/projected/44270a4c-6bad-4213-9db0-de0812258e5a-kube-api-access-srbrm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:32:27.335696 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:27.335662    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"
Sep 17 18:32:27.335883 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:27.335671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8" event={"ID":"44270a4c-6bad-4213-9db0-de0812258e5a","Type":"ContainerDied","Data":"e6db616c6871d8356e6fe010e305e019cf83b346d3a80aae26342d3d55f71163"}
Sep 17 18:32:27.335883 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:27.335767    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6db616c6871d8356e6fe010e305e019cf83b346d3a80aae26342d3d55f71163"
Sep 17 18:32:27.625527 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:27.625504    2581 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada") on node "ip-10-0-133-194.ec2.internal"
Sep 17 18:32:27.668905 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:27.668869    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"csi-f7c53c27b3a4eb8f5b0f73b3272bfa9bdc7cabd18b407b8e56a8bb18204b254e\""
Sep 17 18:32:28.530989 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:28.530933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:28.683835 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:28.683804    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:44.580401 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:32:44.580361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:21.694543 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:33:21.694499    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:39.959175 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:33:39.959144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:41.323958 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:33:41.323926    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:00.187463 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:00.187436    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:34:00.207897 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:00.207876    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:34:03.595156 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:03.595119    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:03.722568 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:03.722533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:20.297833 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:20.297801    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:34:35.319292 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:35.319238    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="176154bb-04c7-4d42-9b08-ceec05c15e4c" containerName="main" containerID="cri-o://6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78" gracePeriod=45
Sep 17 18:34:35.964325 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:35.964302    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:34:36.005952 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.005933    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt22v\" (UniqueName: \"kubernetes.io/projected/176154bb-04c7-4d42-9b08-ceec05c15e4c-kube-api-access-wt22v\") pod \"176154bb-04c7-4d42-9b08-ceec05c15e4c\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") "
Sep 17 18:34:36.006051 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006041    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-home\") pod \"176154bb-04c7-4d42-9b08-ceec05c15e4c\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") "
Sep 17 18:34:36.006095 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006066    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/176154bb-04c7-4d42-9b08-ceec05c15e4c-cabundle-cert\") pod \"176154bb-04c7-4d42-9b08-ceec05c15e4c\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") "
Sep 17 18:34:36.006095 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006081    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176154bb-04c7-4d42-9b08-ceec05c15e4c-tls-certs\") pod \"176154bb-04c7-4d42-9b08-ceec05c15e4c\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") "
Sep 17 18:34:36.006163 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006108    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-tmp-dir\") pod \"176154bb-04c7-4d42-9b08-ceec05c15e4c\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") "
Sep 17 18:34:36.006205 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006178    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-model-cache\") pod \"176154bb-04c7-4d42-9b08-ceec05c15e4c\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") "
Sep 17 18:34:36.006262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-dshm\") pod \"176154bb-04c7-4d42-9b08-ceec05c15e4c\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") "
Sep 17 18:34:36.006262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006228    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-kserve-provision-location\") pod \"176154bb-04c7-4d42-9b08-ceec05c15e4c\" (UID: \"176154bb-04c7-4d42-9b08-ceec05c15e4c\") "
Sep 17 18:34:36.006581 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176154bb-04c7-4d42-9b08-ceec05c15e4c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "176154bb-04c7-4d42-9b08-ceec05c15e4c" (UID: "176154bb-04c7-4d42-9b08-ceec05c15e4c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:34:36.006581 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006546    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "176154bb-04c7-4d42-9b08-ceec05c15e4c" (UID: "176154bb-04c7-4d42-9b08-ceec05c15e4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:36.006745 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.006669    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-home" (OuterVolumeSpecName: "home") pod "176154bb-04c7-4d42-9b08-ceec05c15e4c" (UID: "176154bb-04c7-4d42-9b08-ceec05c15e4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:36.008771 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.008747    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-dshm" (OuterVolumeSpecName: "dshm") pod "176154bb-04c7-4d42-9b08-ceec05c15e4c" (UID: "176154bb-04c7-4d42-9b08-ceec05c15e4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:36.008869 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.008768    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176154bb-04c7-4d42-9b08-ceec05c15e4c-kube-api-access-wt22v" (OuterVolumeSpecName: "kube-api-access-wt22v") pod "176154bb-04c7-4d42-9b08-ceec05c15e4c" (UID: "176154bb-04c7-4d42-9b08-ceec05c15e4c"). InnerVolumeSpecName "kube-api-access-wt22v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:34:36.008869 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.008751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176154bb-04c7-4d42-9b08-ceec05c15e4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "176154bb-04c7-4d42-9b08-ceec05c15e4c" (UID: "176154bb-04c7-4d42-9b08-ceec05c15e4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:34:36.018896 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.018873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "176154bb-04c7-4d42-9b08-ceec05c15e4c" (UID: "176154bb-04c7-4d42-9b08-ceec05c15e4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:36.033743 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.033688    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "176154bb-04c7-4d42-9b08-ceec05c15e4c" (UID: "176154bb-04c7-4d42-9b08-ceec05c15e4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:36.046708 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.046687    2581 generic.go:356] "Generic (PLEG): container finished" podID="176154bb-04c7-4d42-9b08-ceec05c15e4c" containerID="6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78" exitCode=0
Sep 17 18:34:36.046802 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.046735    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:34:36.046802 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.046764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"176154bb-04c7-4d42-9b08-ceec05c15e4c","Type":"ContainerDied","Data":"6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78"}
Sep 17 18:34:36.046876 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.046808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"176154bb-04c7-4d42-9b08-ceec05c15e4c","Type":"ContainerDied","Data":"897af56e9c3f0ab0ae5936195572aa2c3d175919ecfd4c1a49f491e5b9204895"}
Sep 17 18:34:36.046876 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.046832    2581 scope.go:117] "RemoveContainer" containerID="6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78"
Sep 17 18:34:36.055771 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.055756    2581 scope.go:117] "RemoveContainer" containerID="ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10"
Sep 17 18:34:36.063932 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.063909    2581 scope.go:117] "RemoveContainer" containerID="6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78"
Sep 17 18:34:36.064198 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:34:36.064179    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78\": container with ID starting with 6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78 not found: ID does not exist" containerID="6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78"
Sep 17 18:34:36.064267 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.064205    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78"} err="failed to get container status \"6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78\": rpc error: code = NotFound desc = could not find container \"6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78\": container with ID starting with 6010383e60401e1b5333cf1228d41c0bba126135e8584ac6c7ea331c207ccd78 not found: ID does not exist"
Sep 17 18:34:36.064267 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.064220    2581 scope.go:117] "RemoveContainer" containerID="ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10"
Sep 17 18:34:36.064471 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:34:36.064453    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10\": container with ID starting with ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10 not found: ID does not exist" containerID="ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10"
Sep 17 18:34:36.064511 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.064479    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10"} err="failed to get container status \"ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10\": rpc error: code = NotFound desc = could not find container \"ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10\": container with ID starting with ce4e8eb21dd8fd5fddc792e0c5f9a277ea3f8e0072e16b89f17be6785ec84e10 not found: ID does not exist"
Sep 17 18:34:36.073130 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.073109    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:34:36.080649 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.080619    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:34:36.107278 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.107257    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:34:36.107278 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.107279    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:34:36.107397 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.107288    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:34:36.107397 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.107297    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt22v\" (UniqueName: \"kubernetes.io/projected/176154bb-04c7-4d42-9b08-ceec05c15e4c-kube-api-access-wt22v\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:34:36.107397 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.107306    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:34:36.107397 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.107313    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/176154bb-04c7-4d42-9b08-ceec05c15e4c-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:34:36.107397 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.107321    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176154bb-04c7-4d42-9b08-ceec05c15e4c-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:34:36.107397 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:36.107329    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176154bb-04c7-4d42-9b08-ceec05c15e4c-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:34:37.180568 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:37.180536    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="176154bb-04c7-4d42-9b08-ceec05c15e4c" path="/var/lib/kubelet/pods/176154bb-04c7-4d42-9b08-ceec05c15e4c/volumes"
Sep 17 18:34:50.308687 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:50.308658    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:53.768642 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:53.768592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:53.867945 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:34:53.867915    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:00.480077 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:00.479998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:04.941452 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:04.941420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.213685 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:05.213650    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:07.752038 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:07.751991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:07.828055 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:07.828028    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:08.856286 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:08.856225    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:10.716887 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:10.716796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:10.816988 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:10.816941    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:10.917417 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:10.917394    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:15.698173 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:15.698130    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"]
Sep 17 18:35:15.703634 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:15.703605    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-sj7k8"]
Sep 17 18:35:16.047516 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:16.047444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:16.113805 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:16.113778    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:16.209602 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:16.209575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:16.265983 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:16.265939    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.053030 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:17.052994    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.181050 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:17.181023    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44270a4c-6bad-4213-9db0-de0812258e5a" path="/var/lib/kubelet/pods/44270a4c-6bad-4213-9db0-de0812258e5a/volumes"
Sep 17 18:35:18.067488 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:18.067456    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:23.051400 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:23.051364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:23.152252 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:23.152219    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:35.617163 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.617121    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"]
Sep 17 18:35:35.617708 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.617527    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176154bb-04c7-4d42-9b08-ceec05c15e4c" containerName="main"
Sep 17 18:35:35.617708 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.617541    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="176154bb-04c7-4d42-9b08-ceec05c15e4c" containerName="main"
Sep 17 18:35:35.617708 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.617554    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44270a4c-6bad-4213-9db0-de0812258e5a" containerName="storage-initializer"
Sep 17 18:35:35.617708 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.617563    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44270a4c-6bad-4213-9db0-de0812258e5a" containerName="storage-initializer"
Sep 17 18:35:35.617708 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.617579    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176154bb-04c7-4d42-9b08-ceec05c15e4c" containerName="storage-initializer"
Sep 17 18:35:35.617708 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.617588    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="176154bb-04c7-4d42-9b08-ceec05c15e4c" containerName="storage-initializer"
Sep 17 18:35:35.617708 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.617687    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="176154bb-04c7-4d42-9b08-ceec05c15e4c" containerName="main"
Sep 17 18:35:35.617708 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.617706    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="44270a4c-6bad-4213-9db0-de0812258e5a" containerName="storage-initializer"
Sep 17 18:35:35.620233 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.620210    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.623130 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.623100    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:35:35.623327 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.623299    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 18:35:35.623493 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.623326    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 18:35:35.623493 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.623379    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 18:35:35.623493 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.623417    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 18:35:35.623689 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.623379    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-rx7tf\""
Sep 17 18:35:35.630577 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.630545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"]
Sep 17 18:35:35.755775 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.755749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f297b0e-c712-4d40-9a02-478e46536557-cabundle-cert\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.755919 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.755803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-model-cache\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.755919 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.755905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-kserve-provision-location\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.756125 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.755978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-tmp-dir\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.756125 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.756057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-home\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.756125 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.756072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f297b0e-c712-4d40-9a02-478e46536557-tls-certs\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.756125 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.756087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmh45\" (UniqueName: \"kubernetes.io/projected/1f297b0e-c712-4d40-9a02-478e46536557-kube-api-access-pmh45\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.756125 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.756109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-dshm\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.797547 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.797523    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:35.856396 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-model-cache\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856499 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-kserve-provision-location\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856499 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-tmp-dir\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856499 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-home\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856499 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f297b0e-c712-4d40-9a02-478e46536557-tls-certs\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856675 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmh45\" (UniqueName: \"kubernetes.io/projected/1f297b0e-c712-4d40-9a02-478e46536557-kube-api-access-pmh45\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856675 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-dshm\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856675 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f297b0e-c712-4d40-9a02-478e46536557-cabundle-cert\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856824 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-model-cache\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856907 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-kserve-provision-location\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.856987 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.856925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-home\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.857042 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.857023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-tmp-dir\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.857295 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.857276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f297b0e-c712-4d40-9a02-478e46536557-cabundle-cert\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.858759 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.858743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-dshm\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.859114 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.859099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f297b0e-c712-4d40-9a02-478e46536557-tls-certs\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.865378 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.865356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmh45\" (UniqueName: \"kubernetes.io/projected/1f297b0e-c712-4d40-9a02-478e46536557-kube-api-access-pmh45\") pod \"tls-verification-test-kserve-5c4c475676-n8g8z\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:35.932456 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:35.932397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:36.092729 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:36.092698    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:36.210757 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:36.210729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"]
Sep 17 18:35:36.213816 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:35:36.213789    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f297b0e_c712_4d40_9a02_478e46536557.slice/crio-824522e02942461bbfbf0473af30acaeeed5752a4f5f0a2b9fab5bfddcd65e2b WatchSource:0}: Error finding container 824522e02942461bbfbf0473af30acaeeed5752a4f5f0a2b9fab5bfddcd65e2b: Status 404 returned error can't find the container with id 824522e02942461bbfbf0473af30acaeeed5752a4f5f0a2b9fab5bfddcd65e2b
Sep 17 18:35:36.384552 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:36.384522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" event={"ID":"1f297b0e-c712-4d40-9a02-478e46536557","Type":"ContainerStarted","Data":"efdb019c6cc648e8e0020fa9190feb8558548b2d5e2e00e06d05177b60812a46"}
Sep 17 18:35:36.384688 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:36.384576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" event={"ID":"1f297b0e-c712-4d40-9a02-478e46536557","Type":"ContainerStarted","Data":"824522e02942461bbfbf0473af30acaeeed5752a4f5f0a2b9fab5bfddcd65e2b"}
Sep 17 18:35:39.403263 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:39.403226    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f297b0e-c712-4d40-9a02-478e46536557" containerID="efdb019c6cc648e8e0020fa9190feb8558548b2d5e2e00e06d05177b60812a46" exitCode=0
Sep 17 18:35:39.403665 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:39.403299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" event={"ID":"1f297b0e-c712-4d40-9a02-478e46536557","Type":"ContainerDied","Data":"efdb019c6cc648e8e0020fa9190feb8558548b2d5e2e00e06d05177b60812a46"}
Sep 17 18:35:40.411101 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:40.411069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" event={"ID":"1f297b0e-c712-4d40-9a02-478e46536557","Type":"ContainerStarted","Data":"ebf4d75baf7ddba065a84e868dcc72c68bce100382ec5ca7dadaf020afb4e25a"}
Sep 17 18:35:40.434596 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:40.434547    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podStartSLOduration=3.192522875 podStartE2EDuration="5.434531777s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.242008902s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:40.431666155 +0000 UTC m=+3703.356138114" watchObservedRunningTime="2026-09-17 18:35:40.434531777 +0000 UTC m=+3703.359003735"
Sep 17 18:35:40.932682 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:40.932649    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:44.867195 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:44.867162    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:45.932692 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:45.932663    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:35:45.934239 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:45.934214    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:35:46.445504 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:46.445458    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:46.543484 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:46.543457    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:55.932955 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:35:55.932908    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:36:05.933468 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:05.933427    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:36:15.933606 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:15.933565    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:36:25.933317 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:25.933273    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:36:34.730287 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:34.730252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:34.815871 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:34.815839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:35.932936 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:35.932897    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:36:41.015267 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:41.015183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:43.446335 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:43.446300    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:43.682934 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:43.682904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:45.094179 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:45.094145    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:45.933252 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:45.933214    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:36:46.870884 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:46.870851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:46.936557 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:46.936534    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:55.933042 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:36:55.932997    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:37:05.933328 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:05.933279    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:37:07.208262 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:07.208226    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:15.933845 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:15.933806    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:37:17.519647 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:17.519614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:17.615185 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:17.615150    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:25.942647 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:25.942618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:37:25.950308 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:25.950283    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:37:30.434729 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:30.434696    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"]
Sep 17 18:37:30.446804 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:30.446777    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:30.543353 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:30.543320    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:45.457222 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:45.457161    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" containerID="cri-o://ebf4d75baf7ddba065a84e868dcc72c68bce100382ec5ca7dadaf020afb4e25a" gracePeriod=45
Sep 17 18:37:45.943669 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:45.943566    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 17 18:37:46.102983 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.102940    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f297b0e-c712-4d40-9a02-478e46536557" containerID="ebf4d75baf7ddba065a84e868dcc72c68bce100382ec5ca7dadaf020afb4e25a" exitCode=0
Sep 17 18:37:46.103157 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.103004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" event={"ID":"1f297b0e-c712-4d40-9a02-478e46536557","Type":"ContainerDied","Data":"ebf4d75baf7ddba065a84e868dcc72c68bce100382ec5ca7dadaf020afb4e25a"}
Sep 17 18:37:46.203109 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.203086    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:37:46.299767 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.299740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f297b0e-c712-4d40-9a02-478e46536557-tls-certs\") pod \"1f297b0e-c712-4d40-9a02-478e46536557\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") "
Sep 17 18:37:46.299907 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.299775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f297b0e-c712-4d40-9a02-478e46536557-cabundle-cert\") pod \"1f297b0e-c712-4d40-9a02-478e46536557\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") "
Sep 17 18:37:46.299907 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.299852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-dshm\") pod \"1f297b0e-c712-4d40-9a02-478e46536557\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") "
Sep 17 18:37:46.299907 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.299886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-model-cache\") pod \"1f297b0e-c712-4d40-9a02-478e46536557\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") "
Sep 17 18:37:46.300116 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.299910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-home\") pod \"1f297b0e-c712-4d40-9a02-478e46536557\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") "
Sep 17 18:37:46.300116 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.299936    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-tmp-dir\") pod \"1f297b0e-c712-4d40-9a02-478e46536557\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") "
Sep 17 18:37:46.300116 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.299994    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmh45\" (UniqueName: \"kubernetes.io/projected/1f297b0e-c712-4d40-9a02-478e46536557-kube-api-access-pmh45\") pod \"1f297b0e-c712-4d40-9a02-478e46536557\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") "
Sep 17 18:37:46.300116 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.300028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-kserve-provision-location\") pod \"1f297b0e-c712-4d40-9a02-478e46536557\" (UID: \"1f297b0e-c712-4d40-9a02-478e46536557\") "
Sep 17 18:37:46.300312 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.300195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f297b0e-c712-4d40-9a02-478e46536557" (UID: "1f297b0e-c712-4d40-9a02-478e46536557"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:46.300312 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.300210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f297b0e-c712-4d40-9a02-478e46536557-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1f297b0e-c712-4d40-9a02-478e46536557" (UID: "1f297b0e-c712-4d40-9a02-478e46536557"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:46.300413 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.300395    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f297b0e-c712-4d40-9a02-478e46536557-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:37:46.300451 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.300422    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:37:46.300667 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.300636    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-home" (OuterVolumeSpecName: "home") pod "1f297b0e-c712-4d40-9a02-478e46536557" (UID: "1f297b0e-c712-4d40-9a02-478e46536557"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:46.302149 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.302124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-dshm" (OuterVolumeSpecName: "dshm") pod "1f297b0e-c712-4d40-9a02-478e46536557" (UID: "1f297b0e-c712-4d40-9a02-478e46536557"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:46.302263 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.302236    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f297b0e-c712-4d40-9a02-478e46536557-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f297b0e-c712-4d40-9a02-478e46536557" (UID: "1f297b0e-c712-4d40-9a02-478e46536557"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:37:46.302315 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.302283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f297b0e-c712-4d40-9a02-478e46536557-kube-api-access-pmh45" (OuterVolumeSpecName: "kube-api-access-pmh45") pod "1f297b0e-c712-4d40-9a02-478e46536557" (UID: "1f297b0e-c712-4d40-9a02-478e46536557"). InnerVolumeSpecName "kube-api-access-pmh45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:46.320634 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.320604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f297b0e-c712-4d40-9a02-478e46536557" (UID: "1f297b0e-c712-4d40-9a02-478e46536557"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:46.325501 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.325481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f297b0e-c712-4d40-9a02-478e46536557" (UID: "1f297b0e-c712-4d40-9a02-478e46536557"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:46.401631 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.401606    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:37:46.401631 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.401627    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:37:46.401772 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.401635    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:37:46.401772 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.401646    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmh45\" (UniqueName: \"kubernetes.io/projected/1f297b0e-c712-4d40-9a02-478e46536557-kube-api-access-pmh45\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:37:46.401772 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.401654    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f297b0e-c712-4d40-9a02-478e46536557-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:37:46.401772 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:46.401663    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f297b0e-c712-4d40-9a02-478e46536557-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:37:47.109465 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:47.109431    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"
Sep 17 18:37:47.109465 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:47.109454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z" event={"ID":"1f297b0e-c712-4d40-9a02-478e46536557","Type":"ContainerDied","Data":"824522e02942461bbfbf0473af30acaeeed5752a4f5f0a2b9fab5bfddcd65e2b"}
Sep 17 18:37:47.109921 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:47.109521    2581 scope.go:117] "RemoveContainer" containerID="ebf4d75baf7ddba065a84e868dcc72c68bce100382ec5ca7dadaf020afb4e25a"
Sep 17 18:37:47.118791 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:47.118775    2581 scope.go:117] "RemoveContainer" containerID="efdb019c6cc648e8e0020fa9190feb8558548b2d5e2e00e06d05177b60812a46"
Sep 17 18:37:47.133781 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:47.133759    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"]
Sep 17 18:37:47.139029 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:47.139006    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c4c475676-n8g8z"]
Sep 17 18:37:47.180867 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:37:47.180836    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f297b0e-c712-4d40-9a02-478e46536557" path="/var/lib/kubelet/pods/1f297b0e-c712-4d40-9a02-478e46536557/volumes"
Sep 17 18:38:07.610443 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:07.610406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:07.712295 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:07.712267    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:12.640882 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:12.640806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:12.995216 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:12.995191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:14.964766 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:14.964739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:20.541154 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:20.541121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:20.682592 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:20.682562    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:21.012524 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:21.012494    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:21.113509 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:21.113478    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:24.738545 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:24.738515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:26.664241 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:26.664209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:32.291768 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.291730    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"]
Sep 17 18:38:32.292283 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.292262    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="storage-initializer"
Sep 17 18:38:32.292354 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.292287    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="storage-initializer"
Sep 17 18:38:32.292354 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.292323    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main"
Sep 17 18:38:32.292354 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.292333    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main"
Sep 17 18:38:32.292505 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.292422    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f297b0e-c712-4d40-9a02-478e46536557" containerName="main"
Sep 17 18:38:32.294705 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.294685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.297465 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.297440    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 18:38:32.297582 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.297449    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 18:38:32.297582 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.297455    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 18:38:32.297582 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.297559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 18:38:32.297737 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.297580    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-fbrr8\""
Sep 17 18:38:32.297737 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.297608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:38:32.306619 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.306598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"]
Sep 17 18:38:32.339518 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.339498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/870e22af-25d2-442e-a70d-4d1c4cdea5b7-cabundle-cert\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.339621 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.339538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kserve-provision-location\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.339621 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.339555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-home\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.339621 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.339604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7snj9\" (UniqueName: \"kubernetes.io/projected/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kube-api-access-7snj9\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.339745 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.339637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-dshm\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.339745 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.339718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tmp-dir\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.339815 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.339747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-model-cache\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.339815 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.339784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tls-certs\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441100 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kserve-provision-location\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441100 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-home\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441333 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7snj9\" (UniqueName: \"kubernetes.io/projected/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kube-api-access-7snj9\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441333 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-dshm\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441333 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tmp-dir\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441333 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-model-cache\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441333 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tls-certs\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441606 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/870e22af-25d2-442e-a70d-4d1c4cdea5b7-cabundle-cert\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441606 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-home\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441717 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kserve-provision-location\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441717 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tmp-dir\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.441806 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.441770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-model-cache\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.442030 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.442012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/870e22af-25d2-442e-a70d-4d1c4cdea5b7-cabundle-cert\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.443669 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.443645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-dshm\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.443840 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.443824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tls-certs\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.450382 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.450356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7snj9\" (UniqueName: \"kubernetes.io/projected/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kube-api-access-7snj9\") pod \"prestop-hook-test-kserve-7dc56b5d48-vcdcx\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.500163 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.500143    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:32.605736 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.605678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:32.701593 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.701553    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:32.881450 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:32.881376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"]
Sep 17 18:38:32.886524 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:38:32.886492    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870e22af_25d2_442e_a70d_4d1c4cdea5b7.slice/crio-0f7b62b29f1913edebf526be9c60cad609e7725fa3057edcff57f5f3be31867e WatchSource:0}: Error finding container 0f7b62b29f1913edebf526be9c60cad609e7725fa3057edcff57f5f3be31867e: Status 404 returned error can't find the container with id 0f7b62b29f1913edebf526be9c60cad609e7725fa3057edcff57f5f3be31867e
Sep 17 18:38:33.171395 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:33.171290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:33.297405 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:33.297377    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:33.349139 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:33.349102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" event={"ID":"870e22af-25d2-442e-a70d-4d1c4cdea5b7","Type":"ContainerStarted","Data":"967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e"}
Sep 17 18:38:33.349343 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:33.349159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" event={"ID":"870e22af-25d2-442e-a70d-4d1c4cdea5b7","Type":"ContainerStarted","Data":"0f7b62b29f1913edebf526be9c60cad609e7725fa3057edcff57f5f3be31867e"}
Sep 17 18:38:35.360863 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:35.360828    2581 generic.go:356] "Generic (PLEG): container finished" podID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerID="967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e" exitCode=0
Sep 17 18:38:35.361198 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:35.360899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" event={"ID":"870e22af-25d2-442e-a70d-4d1c4cdea5b7","Type":"ContainerDied","Data":"967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e"}
Sep 17 18:38:36.369466 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:36.369388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" event={"ID":"870e22af-25d2-442e-a70d-4d1c4cdea5b7","Type":"ContainerStarted","Data":"d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98"}
Sep 17 18:38:36.400603 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:36.400540    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podStartSLOduration=2.136900618 podStartE2EDuration="4.400518043s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.263617425s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:36.394797339 +0000 UTC m=+3879.319269308" watchObservedRunningTime="2026-09-17 18:38:36.400518043 +0000 UTC m=+3879.324990002"
Sep 17 18:38:36.606483 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:36.606451    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:42.606687 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:42.606648    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:38:42.608337 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:42.608312    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:38:44.651243 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:44.651212    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:48.244352 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:48.244311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:48.342156 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:48.342113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:52.606897 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:38:52.606848    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:39:00.228170 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:00.228136    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:39:00.239816 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:00.239787    2581 scope.go:117] "RemoveContainer" containerID="b75e1b9f2b377491c589ef879e143a6ac357a632b29a42f3ea2a978231f2b3a5"
Sep 17 18:39:00.250167 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:00.250146    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:39:02.606077 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:02.606027    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:39:09.215391 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:09.215354    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:09.274122 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:09.274095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:12.607090 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:12.607038    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:39:22.606930 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:22.606890    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:39:32.606485 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:32.606437    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:39:37.719901 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:37.719871    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:37.811464 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:37.811429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:42.606767 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:42.606684    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:39:43.867261 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:43.867227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:49.535195 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:49.535152    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:49.636182 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:49.636158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:50.745224 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:50.745191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:51.011460 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:51.011372    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:52.606475 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:52.606428    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:39:55.177472 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:39:55.177437    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:02.606189 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:02.606147    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:40:12.606204 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:12.606158    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:40:22.606645 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:22.606600    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:40:32.616607 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:32.616577    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:40:32.630863 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:32.630831    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:40:35.953677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:35.953643    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:36.049805 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:36.049777    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:36.877860 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:36.877821    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"]
Sep 17 18:40:51.900456 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:51.900410    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" containerID="cri-o://d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98" gracePeriod=45
Sep 17 18:40:52.640487 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.640462    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:40:52.739679 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.739650    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tls-certs\") pod \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") "
Sep 17 18:40:52.739839 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.739713    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7snj9\" (UniqueName: \"kubernetes.io/projected/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kube-api-access-7snj9\") pod \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") "
Sep 17 18:40:52.739839 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.739737    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/870e22af-25d2-442e-a70d-4d1c4cdea5b7-cabundle-cert\") pod \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") "
Sep 17 18:40:52.739839 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.739756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-dshm\") pod \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") "
Sep 17 18:40:52.739839 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.739784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-model-cache\") pod \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") "
Sep 17 18:40:52.740105 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.739842    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kserve-provision-location\") pod \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") "
Sep 17 18:40:52.740105 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.739873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tmp-dir\") pod \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") "
Sep 17 18:40:52.740105 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.739901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-home\") pod \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\" (UID: \"870e22af-25d2-442e-a70d-4d1c4cdea5b7\") "
Sep 17 18:40:52.740292 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.740161    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870e22af-25d2-442e-a70d-4d1c4cdea5b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "870e22af-25d2-442e-a70d-4d1c4cdea5b7" (UID: "870e22af-25d2-442e-a70d-4d1c4cdea5b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:40:52.740780 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.740751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-home" (OuterVolumeSpecName: "home") pod "870e22af-25d2-442e-a70d-4d1c4cdea5b7" (UID: "870e22af-25d2-442e-a70d-4d1c4cdea5b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:52.741184 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.741163    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "870e22af-25d2-442e-a70d-4d1c4cdea5b7" (UID: "870e22af-25d2-442e-a70d-4d1c4cdea5b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:52.742456 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.742431    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-dshm" (OuterVolumeSpecName: "dshm") pod "870e22af-25d2-442e-a70d-4d1c4cdea5b7" (UID: "870e22af-25d2-442e-a70d-4d1c4cdea5b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:52.742563 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.742474    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kube-api-access-7snj9" (OuterVolumeSpecName: "kube-api-access-7snj9") pod "870e22af-25d2-442e-a70d-4d1c4cdea5b7" (UID: "870e22af-25d2-442e-a70d-4d1c4cdea5b7"). InnerVolumeSpecName "kube-api-access-7snj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:52.742563 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.742555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "870e22af-25d2-442e-a70d-4d1c4cdea5b7" (UID: "870e22af-25d2-442e-a70d-4d1c4cdea5b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:52.752547 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.752523    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "870e22af-25d2-442e-a70d-4d1c4cdea5b7" (UID: "870e22af-25d2-442e-a70d-4d1c4cdea5b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:52.764560 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.764539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "870e22af-25d2-442e-a70d-4d1c4cdea5b7" (UID: "870e22af-25d2-442e-a70d-4d1c4cdea5b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:52.841118 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.841095    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:40:52.841118 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.841116    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:40:52.841250 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.841128    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:40:52.841250 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.841138    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:40:52.841250 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.841147    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870e22af-25d2-442e-a70d-4d1c4cdea5b7-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:40:52.841250 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.841154    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870e22af-25d2-442e-a70d-4d1c4cdea5b7-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:40:52.841250 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.841162    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7snj9\" (UniqueName: \"kubernetes.io/projected/870e22af-25d2-442e-a70d-4d1c4cdea5b7-kube-api-access-7snj9\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:40:52.841250 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:52.841170    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/870e22af-25d2-442e-a70d-4d1c4cdea5b7-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 17 18:40:53.136904 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.136814    2581 generic.go:356] "Generic (PLEG): container finished" podID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerID="d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98" exitCode=0
Sep 17 18:40:53.136904 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.136877    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"
Sep 17 18:40:53.137506 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.136899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" event={"ID":"870e22af-25d2-442e-a70d-4d1c4cdea5b7","Type":"ContainerDied","Data":"d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98"}
Sep 17 18:40:53.137506 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.136951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" event={"ID":"870e22af-25d2-442e-a70d-4d1c4cdea5b7","Type":"ContainerDied","Data":"0f7b62b29f1913edebf526be9c60cad609e7725fa3057edcff57f5f3be31867e"}
Sep 17 18:40:53.137506 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.136999    2581 scope.go:117] "RemoveContainer" containerID="d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98"
Sep 17 18:40:53.146936 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.146921    2581 scope.go:117] "RemoveContainer" containerID="967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e"
Sep 17 18:40:53.154868 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.154854    2581 scope.go:117] "RemoveContainer" containerID="d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98"
Sep 17 18:40:53.155142 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:40:53.155122    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98\": container with ID starting with d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98 not found: ID does not exist" containerID="d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98"
Sep 17 18:40:53.155202 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.155150    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98"} err="failed to get container status \"d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98\": rpc error: code = NotFound desc = could not find container \"d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98\": container with ID starting with d7b7a1e9ae3ec70e27f507e63940dcb3962ef2e0f2283882b0907661b733cb98 not found: ID does not exist"
Sep 17 18:40:53.155202 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.155167    2581 scope.go:117] "RemoveContainer" containerID="967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e"
Sep 17 18:40:53.155422 ip-10-0-133-194 kubenswrapper[2581]: E0917 18:40:53.155404    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e\": container with ID starting with 967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e not found: ID does not exist" containerID="967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e"
Sep 17 18:40:53.155476 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.155431    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e"} err="failed to get container status \"967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e\": rpc error: code = NotFound desc = could not find container \"967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e\": container with ID starting with 967a108c0d1da2853d6de8c61f2562c40279ce1c4900760f9a094e9afd58378e not found: ID does not exist"
Sep 17 18:40:53.161763 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.161740    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"]
Sep 17 18:40:53.166176 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.166154    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx"]
Sep 17 18:40:53.181515 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.181490    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" path="/var/lib/kubelet/pods/870e22af-25d2-442e-a70d-4d1c4cdea5b7/volumes"
Sep 17 18:40:53.617614 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:40:53.617585    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7dc56b5d48-vcdcx" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:41:13.620177 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:41:13.620101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:13.719104 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:41:13.719069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:26.834622 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:41:26.834585    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:33.849632 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:41:33.849595    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:39.469699 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:41:39.469659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:39.532644 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:41:39.532612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:00.481575 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:00.481540    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:02.262149 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:02.262115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:02.340713 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:02.340685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:06.829233 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:06.829202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:09.084041 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:09.083999    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:12.927528 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:12.927488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:13.021614 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:13.021587    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:26.687855 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:26.687821    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/0.log"
Sep 17 18:42:26.690587 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:26.690563    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/1.log"
Sep 17 18:42:27.466801 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:27.466773    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/0.log"
Sep 17 18:42:27.469845 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:27.469821    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/1.log"
Sep 17 18:42:28.312707 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:28.312682    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7mxrx_069af643-5cf4-457e-9168-51b5e57908fc/authorino/0.log"
Sep 17 18:42:28.365457 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:28.365437    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9bw7q_dde2aab7-9294-4560-bf14-fb580f4f1268/kuadrant-console-plugin/0.log"
Sep 17 18:42:28.452565 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:28.452541    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dpc59_a87d7776-a29c-4440-98cc-22b7fcfd8b31/limitador/0.log"
Sep 17 18:42:34.073122 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.073086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:34.200921 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.200890    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mv6cb/must-gather-szg9r"]
Sep 17 18:42:34.201280 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.201266    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main"
Sep 17 18:42:34.201327 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.201281    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main"
Sep 17 18:42:34.201327 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.201297    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="storage-initializer"
Sep 17 18:42:34.201327 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.201304    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="storage-initializer"
Sep 17 18:42:34.201427 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.201379    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="870e22af-25d2-442e-a70d-4d1c4cdea5b7" containerName="main"
Sep 17 18:42:34.203554 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.203537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6cb/must-gather-szg9r"
Sep 17 18:42:34.206328 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.206306    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mv6cb\"/\"default-dockercfg-nvlfk\""
Sep 17 18:42:34.206465 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.206411    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mv6cb\"/\"kube-root-ca.crt\""
Sep 17 18:42:34.206624 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.206589    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mv6cb\"/\"openshift-service-ca.crt\""
Sep 17 18:42:34.215460 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.215435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6cb/must-gather-szg9r"]
Sep 17 18:42:34.220378 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.220354    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:34.320214 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.320185    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:34.328316 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.328253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrjn\" (UniqueName: \"kubernetes.io/projected/dc7a1082-79cd-4aad-9692-0214e63e90a7-kube-api-access-gjrjn\") pod \"must-gather-szg9r\" (UID: \"dc7a1082-79cd-4aad-9692-0214e63e90a7\") " pod="openshift-must-gather-mv6cb/must-gather-szg9r"
Sep 17 18:42:34.328316 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.328299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc7a1082-79cd-4aad-9692-0214e63e90a7-must-gather-output\") pod \"must-gather-szg9r\" (UID: \"dc7a1082-79cd-4aad-9692-0214e63e90a7\") " pod="openshift-must-gather-mv6cb/must-gather-szg9r"
Sep 17 18:42:34.428802 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.428775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrjn\" (UniqueName: \"kubernetes.io/projected/dc7a1082-79cd-4aad-9692-0214e63e90a7-kube-api-access-gjrjn\") pod \"must-gather-szg9r\" (UID: \"dc7a1082-79cd-4aad-9692-0214e63e90a7\") " pod="openshift-must-gather-mv6cb/must-gather-szg9r"
Sep 17 18:42:34.428916 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.428810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc7a1082-79cd-4aad-9692-0214e63e90a7-must-gather-output\") pod \"must-gather-szg9r\" (UID: \"dc7a1082-79cd-4aad-9692-0214e63e90a7\") " pod="openshift-must-gather-mv6cb/must-gather-szg9r"
Sep 17 18:42:34.429180 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.429160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc7a1082-79cd-4aad-9692-0214e63e90a7-must-gather-output\") pod \"must-gather-szg9r\" (UID: \"dc7a1082-79cd-4aad-9692-0214e63e90a7\") " pod="openshift-must-gather-mv6cb/must-gather-szg9r"
Sep 17 18:42:34.441677 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.441644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrjn\" (UniqueName: \"kubernetes.io/projected/dc7a1082-79cd-4aad-9692-0214e63e90a7-kube-api-access-gjrjn\") pod \"must-gather-szg9r\" (UID: \"dc7a1082-79cd-4aad-9692-0214e63e90a7\") " pod="openshift-must-gather-mv6cb/must-gather-szg9r"
Sep 17 18:42:34.514540 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.514509    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6cb/must-gather-szg9r"
Sep 17 18:42:34.787516 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.787491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6cb/must-gather-szg9r"]
Sep 17 18:42:34.789113 ip-10-0-133-194 kubenswrapper[2581]: W0917 18:42:34.789084    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc7a1082_79cd_4aad_9692_0214e63e90a7.slice/crio-7d5a1bf0f7c31c15ba7c71c88c8fb0c03d67c382b048f1c2f98d61fb6dc2f56b WatchSource:0}: Error finding container 7d5a1bf0f7c31c15ba7c71c88c8fb0c03d67c382b048f1c2f98d61fb6dc2f56b: Status 404 returned error can't find the container with id 7d5a1bf0f7c31c15ba7c71c88c8fb0c03d67c382b048f1c2f98d61fb6dc2f56b
Sep 17 18:42:34.791139 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:34.791119    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:42:35.676590 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:35.676556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6cb/must-gather-szg9r" event={"ID":"dc7a1082-79cd-4aad-9692-0214e63e90a7","Type":"ContainerStarted","Data":"7d5a1bf0f7c31c15ba7c71c88c8fb0c03d67c382b048f1c2f98d61fb6dc2f56b"}
Sep 17 18:42:36.521257 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:36.521230    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:36.687235 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:36.687191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6cb/must-gather-szg9r" event={"ID":"dc7a1082-79cd-4aad-9692-0214e63e90a7","Type":"ContainerStarted","Data":"c1fa9f2521794d81e94820ce1ac3e949a0e1c08aa8660fc50aa16de427340cee"}
Sep 17 18:42:36.687713 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:36.687250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6cb/must-gather-szg9r" event={"ID":"dc7a1082-79cd-4aad-9692-0214e63e90a7","Type":"ContainerStarted","Data":"7662ffed498e50d4d1b59450caf3a57569a58971a75b910d5c164e14cc09154f"}
Sep 17 18:42:36.703839 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:36.703778    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mv6cb/must-gather-szg9r" podStartSLOduration=1.870629864 podStartE2EDuration="2.703758019s" totalImagesPullingTime="833.128155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:36.702897896 +0000 UTC m=+4119.627369856" watchObservedRunningTime="2026-09-17 18:42:36.703758019 +0000 UTC m=+4119.628229980"
Sep 17 18:42:37.191204 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:37.191160    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n5zs2_243ab975-ee79-47da-9259-405c837db145/global-pull-secret-syncer/0.log"
Sep 17 18:42:37.269466 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:37.269436    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log"
Sep 17 18:42:37.388947 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:37.388911    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-194.ec2.internal_2c0c5f3cddfb693025739ba654373f8f/haproxy/0.log"
Sep 17 18:42:40.290430 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:40.290340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:40.353839 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:40.353775    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:42.582998 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:42.582501    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7mxrx_069af643-5cf4-457e-9168-51b5e57908fc/authorino/0.log"
Sep 17 18:42:42.678381 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:42.678341    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9bw7q_dde2aab7-9294-4560-bf14-fb580f4f1268/kuadrant-console-plugin/0.log"
Sep 17 18:42:42.796685 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:42.796642    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dpc59_a87d7776-a29c-4440-98cc-22b7fcfd8b31/limitador/0.log"
Sep 17 18:42:44.222624 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.222589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-k4shc_ee1b6649-932e-404d-a2e8-8960eba66ae7/cluster-monitoring-operator/0.log"
Sep 17 18:42:44.262056 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.262023    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-qpqth_b3ac56f6-e1a0-4358-a163-9ea0f22a91a8/kube-state-metrics/0.log"
Sep 17 18:42:44.284849 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.284782    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-qpqth_b3ac56f6-e1a0-4358-a163-9ea0f22a91a8/kube-rbac-proxy-main/0.log"
Sep 17 18:42:44.313825 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.313775    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-qpqth_b3ac56f6-e1a0-4358-a163-9ea0f22a91a8/kube-rbac-proxy-self/0.log"
Sep 17 18:42:44.346524 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.346468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85cc868f54-p4pmd_4091c712-43c0-43a7-a915-c3f4ff2c8276/metrics-server/0.log"
Sep 17 18:42:44.568868 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.568784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xshpf_43182a8d-c31e-4b95-94e5-9e94ca06d6af/node-exporter/0.log"
Sep 17 18:42:44.594538 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.594505    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xshpf_43182a8d-c31e-4b95-94e5-9e94ca06d6af/kube-rbac-proxy/0.log"
Sep 17 18:42:44.618445 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.618410    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xshpf_43182a8d-c31e-4b95-94e5-9e94ca06d6af/init-textfile/0.log"
Sep 17 18:42:44.653815 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.653775    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-xj5qz_08297acf-4eb9-44de-9039-d828bc27d430/kube-rbac-proxy-main/0.log"
Sep 17 18:42:44.682562 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.682528    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-xj5qz_08297acf-4eb9-44de-9039-d828bc27d430/kube-rbac-proxy-self/0.log"
Sep 17 18:42:44.718911 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.718878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-xj5qz_08297acf-4eb9-44de-9039-d828bc27d430/openshift-state-metrics/0.log"
Sep 17 18:42:44.988187 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:44.988159    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-l466x_cd28df02-e0f6-401d-ab7f-61f50bc4ed39/prometheus-operator/0.log"
Sep 17 18:42:45.008128 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:45.008082    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-l466x_cd28df02-e0f6-401d-ab7f-61f50bc4ed39/kube-rbac-proxy/0.log"
Sep 17 18:42:45.149466 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:45.149426    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f55c86c89-2xk5g_c23ba056-4962-463d-8c1e-590f4bfbefcb/thanos-query/0.log"
Sep 17 18:42:45.172566 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:45.172535    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f55c86c89-2xk5g_c23ba056-4962-463d-8c1e-590f4bfbefcb/kube-rbac-proxy-web/0.log"
Sep 17 18:42:45.198201 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:45.198167    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f55c86c89-2xk5g_c23ba056-4962-463d-8c1e-590f4bfbefcb/kube-rbac-proxy/0.log"
Sep 17 18:42:45.225535 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:45.225509    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f55c86c89-2xk5g_c23ba056-4962-463d-8c1e-590f4bfbefcb/prom-label-proxy/0.log"
Sep 17 18:42:45.252894 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:45.252814    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f55c86c89-2xk5g_c23ba056-4962-463d-8c1e-590f4bfbefcb/kube-rbac-proxy-rules/0.log"
Sep 17 18:42:45.277287 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:45.277257    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f55c86c89-2xk5g_c23ba056-4962-463d-8c1e-590f4bfbefcb/kube-rbac-proxy-metrics/0.log"
Sep 17 18:42:47.257180 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.257143    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"]
Sep 17 18:42:47.263318 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.263285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.272608 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.272579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"]
Sep 17 18:42:47.369101 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.369069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-lib-modules\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.369259 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.369205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-sys\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.369364 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.369340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-podres\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.369403 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.369380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-proc\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.369461 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.369444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsg4n\" (UniqueName: \"kubernetes.io/projected/a9359c50-8f2c-4557-9b41-065b511b9be7-kube-api-access-rsg4n\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.470801 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.470755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-lib-modules\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.470982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.470837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-sys\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.470982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.470902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-podres\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.470982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.470925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-proc\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.470982 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.470959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-lib-modules\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.471165 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.470987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsg4n\" (UniqueName: \"kubernetes.io/projected/a9359c50-8f2c-4557-9b41-065b511b9be7-kube-api-access-rsg4n\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.471165 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.470960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-sys\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.471165 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.471013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-proc\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.471305 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.471233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9359c50-8f2c-4557-9b41-065b511b9be7-podres\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.474556 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.474536    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b7c9d8857-hzgbz_a425cd1c-90e6-491d-a77e-a62a5fa87354/console/0.log"
Sep 17 18:42:47.480138 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.480110    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsg4n\" (UniqueName: \"kubernetes.io/projected/a9359c50-8f2c-4557-9b41-065b511b9be7-kube-api-access-rsg4n\") pod \"perf-node-gather-daemonset-mb45d\" (UID: \"a9359c50-8f2c-4557-9b41-065b511b9be7\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.528870 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.528800    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-8g78l_a5fdaada-7867-4a8f-b7ee-9971ef25fbf7/download-server/0.log"
Sep 17 18:42:47.578437 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.578402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:47.881309 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:47.881272    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"]
Sep 17 18:42:48.103421 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:48.103388    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-xb2s5_5230ba66-378a-4c58-a52b-df47433c0377/volume-data-source-validator/0.log"
Sep 17 18:42:48.794472 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:48.794436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d" event={"ID":"a9359c50-8f2c-4557-9b41-065b511b9be7","Type":"ContainerStarted","Data":"cea3873b4b7eba686a3e1a35d98ef9fca1b14d851ebf2549565bfd65c3e413c6"}
Sep 17 18:42:48.794870 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:48.794495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d" event={"ID":"a9359c50-8f2c-4557-9b41-065b511b9be7","Type":"ContainerStarted","Data":"e69b4dae449c5fc63d29884573b686bfa461a113936a71a66651e8a5a71b97ae"}
Sep 17 18:42:48.794870 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:48.794533    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:48.812242 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:48.812196    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d" podStartSLOduration=1.812181582 podStartE2EDuration="1.812181582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:48.810034927 +0000 UTC m=+4131.734506886" watchObservedRunningTime="2026-09-17 18:42:48.812181582 +0000 UTC m=+4131.736653541"
Sep 17 18:42:49.161135 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:49.161059    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s2zzx_1dc2dde7-c0ad-4ce1-995e-a94870a19542/dns/0.log"
Sep 17 18:42:49.183428 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:49.183400    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s2zzx_1dc2dde7-c0ad-4ce1-995e-a94870a19542/kube-rbac-proxy/0.log"
Sep 17 18:42:49.213860 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:49.213809    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c44np_022180cf-1016-4aa8-ad6a-9b6d20adfeba/dns-node-resolver/0.log"
Sep 17 18:42:49.836689 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:49.836660    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-frlbs_388ecc81-7714-4160-adbb-e4e68c6f66ef/node-ca/0.log"
Sep 17 18:42:50.860535 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:50.860502    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/0.log"
Sep 17 18:42:50.864610 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:50.864584    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/1.log"
Sep 17 18:42:51.387637 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:51.387604    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h5r2d_2fdaa930-c54c-49fe-ba86-664069237f41/serve-healthcheck-canary/0.log"
Sep 17 18:42:51.904511 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:51.904482    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-b47t9_4575b36c-ec44-42ed-b49f-3cd2920ed4d5/insights-operator/0.log"
Sep 17 18:42:51.907207 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:51.907178    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-b47t9_4575b36c-ec44-42ed-b49f-3cd2920ed4d5/insights-operator/1.log"
Sep 17 18:42:52.005120 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:52.005094    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nwps2_e71af6a9-06ca-4b00-be3d-c179616ae9ec/kube-rbac-proxy/0.log"
Sep 17 18:42:52.026551 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:52.026527    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nwps2_e71af6a9-06ca-4b00-be3d-c179616ae9ec/exporter/0.log"
Sep 17 18:42:52.049101 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:52.049074    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nwps2_e71af6a9-06ca-4b00-be3d-c179616ae9ec/extractor/0.log"
Sep 17 18:42:54.813300 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:54.813266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-mb45d"
Sep 17 18:42:55.024571 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:55.024540    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-ff6d497d7-fj4pr_f967781f-96ca-4b8c-abed-3ddfb3616ddb/manager/0.log"
Sep 17 18:42:56.407735 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:56.407695    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/0.log"
Sep 17 18:42:56.412771 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:56.412743    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/1.log"
Sep 17 18:42:57.613136 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:57.613049    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/0.log"
Sep 17 18:42:57.617134 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:42:57.617082    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-vq6jg_67f34990-ab55-4d22-b575-ae36e8990569/istio-proxy/1.log"
Sep 17 18:43:00.710187 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:00.710143    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9zcnt_d765f4f6-1b2b-4c0a-b8e9-5228eb337b1e/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:01.578570 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:01.578537    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7cfs2_cd8188a2-f9f7-405a-a26d-40551aa263df/migrator/0.log"
Sep 17 18:43:01.600655 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:01.600630    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7cfs2_cd8188a2-f9f7-405a-a26d-40551aa263df/graceful-termination/0.log"
Sep 17 18:43:01.943344 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:01.943262    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-4k49d_c894143e-e9a8-49a9-9e93-0de501d60d3b/kube-storage-version-migrator-operator/1.log"
Sep 17 18:43:01.944271 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:01.944247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-4k49d_c894143e-e9a8-49a9-9e93-0de501d60d3b/kube-storage-version-migrator-operator/0.log"
Sep 17 18:43:03.206272 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.206247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vtvtb_c87361b4-40c1-45cd-a813-271dfe0802d4/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:43:03.230476 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.230428    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vtvtb_c87361b4-40c1-45cd-a813-271dfe0802d4/egress-router-binary-copy/0.log"
Sep 17 18:43:03.254538 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.254516    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vtvtb_c87361b4-40c1-45cd-a813-271dfe0802d4/cni-plugins/0.log"
Sep 17 18:43:03.276651 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.276632    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vtvtb_c87361b4-40c1-45cd-a813-271dfe0802d4/bond-cni-plugin/0.log"
Sep 17 18:43:03.298452 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.298431    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vtvtb_c87361b4-40c1-45cd-a813-271dfe0802d4/routeoverride-cni/0.log"
Sep 17 18:43:03.321002 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.320945    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vtvtb_c87361b4-40c1-45cd-a813-271dfe0802d4/whereabouts-cni-bincopy/0.log"
Sep 17 18:43:03.343447 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.343426    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vtvtb_c87361b4-40c1-45cd-a813-271dfe0802d4/whereabouts-cni/0.log"
Sep 17 18:43:03.423814 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.423767    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lg444_f8b7d5a8-97c2-4370-9cba-3a0ae2133bff/kube-multus/0.log"
Sep 17 18:43:03.597685 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.597610    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tdmfn_3b795638-8c73-4961-9d78-d850444de74c/network-metrics-daemon/0.log"
Sep 17 18:43:03.643749 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:03.643706    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tdmfn_3b795638-8c73-4961-9d78-d850444de74c/kube-rbac-proxy/0.log"
Sep 17 18:43:04.690818 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:04.690788    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-controller/0.log"
Sep 17 18:43:04.709629 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:04.709604    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/0.log"
Sep 17 18:43:04.709774 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:04.709602    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/ovn-acl-logging/1.log"
Sep 17 18:43:04.731906 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:04.731885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/kube-rbac-proxy-node/0.log"
Sep 17 18:43:04.754197 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:04.754176    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:43:04.773889 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:04.773872    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/northd/0.log"
Sep 17 18:43:04.796911 ip-10-0-133-194 kubenswrapper[2581]: I0917 18:43:04.796893    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbjdm_89ba270d-bdb8-4eb6-ae19-a569f740c1e8/nbdb/0.log"