Sep 18 21:50:21.017518 ip-10-0-130-79 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 21:50:21.462365 ip-10-0-130-79 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:50:21.462365 ip-10-0-130-79 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 21:50:21.462365 ip-10-0-130-79 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:50:21.462365 ip-10-0-130-79 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 21:50:21.462365 ip-10-0-130-79 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:50:21.463330 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.463233    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 21:50:21.467492 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467473    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:21.467492 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467488    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:21.467492 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467493    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:21.467492 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467497    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467502    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467507    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467511    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467515    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467519    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467523    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467526    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467531    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467534    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467538    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467542    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467551    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467555    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467559    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467564    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467567    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467571    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467575    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:21.467736 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467579    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467583    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467587    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467591    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467595    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467599    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467603    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467607    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467611    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467615    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467619    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467623    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467627    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467631    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467636    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467641    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467645    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467650    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467654    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467658    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:21.468545 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467662    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467666    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467671    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467675    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467682    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467688    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467694    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467699    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467704    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467709    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467714    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467722    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467727    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467732    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467736    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467740    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467744    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467748    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467752    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:21.469344 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467756    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467760    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467765    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467771    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467778    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467783    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467788    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467792    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467798    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467803    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467807    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467812    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467816    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467820    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467824    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467828    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467834    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467838    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467843    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467847    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:21.469823 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467851    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467856    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467861    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467866    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467870    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467875    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467879    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467883    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467887    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467892    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467896    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467900    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467904    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467909    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467913    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467918    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467922    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:21.470326 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.467926    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:21.471109 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471093    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:21.471109 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471109    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471114    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471117    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471120    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471123    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471126    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471129    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471132    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471134    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471137    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471140    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471143    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471145    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471148    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471151    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471153    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471156    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471158    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471161    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:21.471166 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471163    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471166    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471169    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471171    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471174    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471177    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471180    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471185    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471189    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471191    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471194    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471197    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471200    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471203    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471205    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471208    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471210    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471213    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471215    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:21.471657 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471220    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471224    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471226    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471229    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471232    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471234    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471237    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471239    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471242    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471244    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471247    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471249    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471252    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471255    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471258    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471261    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471277    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471282    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471285    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471288    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:21.472130 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471291    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471294    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471296    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471299    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471303    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471305    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471308    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471310    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471313    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471316    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471318    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471321    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471323    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471327    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471331    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471333    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471336    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471338    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471341    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471343    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:21.472638 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471346    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471348    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471351    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471353    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471358    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471361    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471363    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471367    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471370    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471372    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471375    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471378    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471380    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471383    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471385    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471388    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471391    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471393    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471396    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471398    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:21.473128 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471763    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471768    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471771    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471773    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471776    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471778    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471781    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471783    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471786    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471789    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471791    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471793    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471796    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471798    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471801    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471803    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471806    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471809    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471812    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471817    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:21.473630 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471820    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471824    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471827    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471830    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471833    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471836    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471839    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471841    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471844    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471847    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471849    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471852    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471854    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471857    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471860    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471862    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471864    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471867    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471870    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471872    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:21.474117 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471875    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471877    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471880    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471882    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471885    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471887    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471890    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471892    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471895    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471899    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471901    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471904    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471907    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471909    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471912    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471914    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471917    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471919    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471922    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471925    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:21.474627 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471927    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471930    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471933    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471935    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471938    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471941    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471944    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471947    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471950    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471952    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471955    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471957    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471960    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471963    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471965    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471968    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471970    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471973    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471976    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471979    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:21.475115 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471982    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471985    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471988    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471990    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471993    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471996    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.471998    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472001    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472003    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472006    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472010    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472013    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472016    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472019    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472021    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472024    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472026    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472029    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472031    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:21.475607 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472104    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472146    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472152    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472156    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472161    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472164    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472168    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472172    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472175    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472178    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472184    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472188    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472191    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472194    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472198    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472201    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472205    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472208    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472213    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472216    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472219    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472222    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472226    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472234    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 21:50:21.476099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472237    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472240    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472243    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472246    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472249    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472252    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472275    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472280    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472286    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472289    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472292    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472295    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472298    2564 flags.go:64] FLAG: --enable-server="true"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472302    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472306    2564 flags.go:64] FLAG: --event-burst="100"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472309    2564 flags.go:64] FLAG: --event-qps="50"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472311    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472314    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472319    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472324    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472327    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472330    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472333    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472335    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472338    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 21:50:21.476715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472342    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472345    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472348    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472351    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472353    2564 flags.go:64] FLAG: --feature-gates=""
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472357    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472360    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472363    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472366    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472369    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472372    2564 flags.go:64] FLAG: --help="false"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472375    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472378    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472381    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472384    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472387    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472390    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472393    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472396    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472399    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472402    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472405    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472408    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472411    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 18 21:50:21.477431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472414    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472418    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472422    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472425    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472428    2564 flags.go:64] FLAG: --lock-file=""
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472431    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472434    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472437    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472442    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472446    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472448    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472451    2564 flags.go:64] FLAG: --logging-format="text"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472454    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472457    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472460    2564 flags.go:64] FLAG: --manifest-url=""
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472463    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472467    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472470    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472474    2564 flags.go:64] FLAG: --max-pods="110"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472477    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472480    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472483    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472486    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472489    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472492    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 21:50:21.478040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472495    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472501    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472504    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472507    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472510    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472513    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472519    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472522    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472525    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472532    2564 flags.go:64] FLAG: --port="10250"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472534    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472538    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0af14a86195ad2163"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472541    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472544    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472547    2564 flags.go:64] FLAG: --register-node="true"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472550    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472553    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472557    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472560    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472563    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472566    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472569    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472572    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472575    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472578    2564 flags.go:64] FLAG: --runonce="false"
Sep 18 21:50:21.478672 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472581    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472584    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472587    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472590    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472593    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472596    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472599    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472602    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472605    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472607    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472610    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472613    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472616    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472619    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472622    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472627    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472633    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472636    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472640    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472643    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472646    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472649    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472652    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472655    2564 flags.go:64] FLAG: --v="2"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472659    2564 flags.go:64] FLAG: --version="false"
Sep 18 21:50:21.479335 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472664    2564 flags.go:64] FLAG: --vmodule=""
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472667    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.472670    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472785    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472789    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472792    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472795    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472797    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472800    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472803    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472806    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472809    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472811    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472814    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472817    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472820    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472822    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472825    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472828    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472831    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:21.480006 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472833    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472836    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472838    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472844    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472846    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472850    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472854    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472857    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472860    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472863    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472866    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472868    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472871    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472874    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472877    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472880    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472882    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472885    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472888    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472890    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:21.480524 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472893    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472896    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472899    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472901    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472904    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472907    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472909    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472912    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472915    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472917    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472920    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472923    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472926    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472928    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472931    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472937    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472939    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472942    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472945    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472947    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:21.481031 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472950    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472952    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472957    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472960    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472963    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472966    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472970    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472973    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472975    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472978    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472981    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472983    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472986    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472989    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472991    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472994    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472996    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.472999    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473002    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473004    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:21.481554 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473007    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473010    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473012    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473015    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473018    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473021    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473024    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473029    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473031    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473034    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473036    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473041    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473044    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473048    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473051    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473054    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473057    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473060    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473063    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:21.482046 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473067    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473070    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.473073    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.473783    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.481208    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.481221    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.481233    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.481239    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481304    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481309    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481313    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481316    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481319    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481322    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:21.482535 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481326    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481331    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481334    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481337    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481340    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481343    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481345    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481348    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481350    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481353    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481356    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481359    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481361    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481364    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481367    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481369    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481372    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481375    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481378    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481381    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:21.482901 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481384    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481387    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481389    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481392    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481396    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481399    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481401    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481404    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481407    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481409    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481412    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481415    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481417    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481420    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481422    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481425    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481427    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481430    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481432    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481435    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:21.483445 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481438    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481441    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481443    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481446    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481448    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481451    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481453    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481456    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481458    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481461    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481464    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481467    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481472    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481475    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481479    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481483    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481487    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481490    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481493    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:21.483931 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481496    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481499    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481502    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481505    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481508    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481510    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481513    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481516    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481518    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481521    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481523    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481526    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481528    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481531    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481533    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481536    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481539    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481541    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481544    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481546    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:21.484538 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481549    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481552    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481554    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481557    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481561    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481565    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481567    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481570    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481573    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481576    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481578    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481581    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481583    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481586    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:21.485018 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.481591    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481696    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481701    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481704    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481707    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481710    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481712    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481715    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481718    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481720    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481723    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481726    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481730    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481733    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481736    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481739    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481741    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481744    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481746    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481749    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:21.485392 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481752    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481755    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481757    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481760    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481763    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481766    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481770    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481773    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481776    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481779    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481782    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481784    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481787    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481789    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481792    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481795    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481797    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481800    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481802    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:21.485885 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481805    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481808    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481811    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481813    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481816    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481818    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481821    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481824    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481826    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481829    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481831    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481834    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481836    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481839    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481842    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481844    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481864    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481867    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481871    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481874    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481877    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:21.486381 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481880    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481883    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481885    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481887    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481890    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481893    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481895    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481898    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481901    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481905    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481908    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481910    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481913    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481916    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481918    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481921    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481924    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481926    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481929    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:21.486890 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481932    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481934    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481936    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481939    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481942    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481945    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481947    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481950    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481952    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481955    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481957    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481960    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481963    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481965    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481968    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481971    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481974    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481976    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481979    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481981    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:21.487373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:21.481984    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:21.487859 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.481988    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:50:21.487859 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.482076    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 21:50:21.487859 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.484424    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 21:50:21.487859 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.485502    2564 server.go:1019] "Starting client certificate rotation"
Sep 18 21:50:21.487859 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.485605    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 21:50:21.487859 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.485634    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 21:50:21.488386 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.488371    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 21:50:21.491549 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.491534    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 21:50:21.502018 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.502001    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 18 21:50:21.504453 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.504253    2564 log.go:25] "Validated CRI v1 image API"
Sep 18 21:50:21.505441 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.505425    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 21:50:21.508621 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.508604    2564 fs.go:135] Filesystem UUIDs: map[42a318b5-de49-4b83-9d7f-d07c5c3f30cf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7e5ec87c-daf8-4d59-b6d6-598bf5e66b6d:/dev/nvme0n1p3]
Sep 18 21:50:21.508693 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.508620    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 21:50:21.513570 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.513466    2564 manager.go:216] Machine: {Timestamp:2026-09-18 21:50:21.512049145 +0000 UTC m=+0.368704788 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100641 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2c4252101b04e5566e32462d74a0 SystemUUID:ec2a2c42-5210-1b04-e556-6e32462d74a0 BootID:4e3003bf-8b66-443b-bac6-c2014ccfa6ec Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c7:43:b1:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c7:43:b1:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:de:db:92:dd:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 21:50:21.513570 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.513564    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 21:50:21.513678 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.513632    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 21:50:21.517043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.517024    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 21:50:21.517162 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.517046    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 21:50:21.518542 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.518532    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 21:50:21.518582 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.518545    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 21:50:21.518582 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.518552    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 21:50:21.519388 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.519376    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 21:50:21.519469 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.519457    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 21:50:21.524209 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.524198    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 21:50:21.524350 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.524341    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 21:50:21.524382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.524373    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 21:50:21.524409 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.524389    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 21:50:21.524409 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.524406    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 21:50:21.524468 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.524415    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 18 21:50:21.524468 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.524423    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 21:50:21.524714 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.524703    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 21:50:21.524749 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.524722    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 21:50:21.527111 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.527094    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 21:50:21.529808 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.529796    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 21:50:21.529860 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.529825    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 21:50:21.531140 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531130    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 21:50:21.531181 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531145    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 21:50:21.531181 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531151    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 21:50:21.531181 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531155    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 21:50:21.531181 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531164    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 21:50:21.531181 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531169    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 21:50:21.531181 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531182    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 21:50:21.531345 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531187    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 21:50:21.531345 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531194    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 21:50:21.531345 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531199    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 21:50:21.531345 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531212    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 21:50:21.531345 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531217    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 21:50:21.531741 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.531726    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 21:50:21.532116 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.532106    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 21:50:21.532148 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.532117    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 21:50:21.535822 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.535804    2564 server.go:1268] "Started kubelet"
Sep 18 21:50:21.535905 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.535869    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 21:50:21.535960 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.535888    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 21:50:21.536006 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.535967    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 21:50:21.536541 ip-10-0-130-79 systemd[1]: Started Kubernetes Kubelet.
Sep 18 21:50:21.537594 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.537494    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 21:50:21.538312 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.538224    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-79.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 21:50:21.542627 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.542603    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 21:50:21.543236 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.543215    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 21:50:21.543350 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.543303    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 21:50:21.547384 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.547363    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 21:50:21.548649 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.547551    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18d688b702ec39fc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-09-18 21:50:21.535779324 +0000 UTC m=+0.392434967,LastTimestamp:2026-09-18 21:50:21.535779324 +0000 UTC m=+0.392434967,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}"
Sep 18 21:50:21.550042 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.550020    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 21:50:21.550042 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.550027    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 21:50:21.550200 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.550098    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:21.550848 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.550733    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 21:50:21.550848 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.550752    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 21:50:21.550848 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.550819    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 21:50:21.550848 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.550828    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 21:50:21.551086 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.551072    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 21:50:21.551231 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.551210    2564 factory.go:55] Registering systemd factory
Sep 18 21:50:21.551231 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.551233    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 18 21:50:21.551425 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.551408    2564 factory.go:153] Registering CRI-O factory
Sep 18 21:50:21.551481 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.551459    2564 factory.go:223] Registration of the crio container factory successfully
Sep 18 21:50:21.551530 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.551507    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 21:50:21.551530 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.551525    2564 factory.go:103] Registering Raw factory
Sep 18 21:50:21.551633 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.551539    2564 manager.go:1196] Started watching for new ooms in manager
Sep 18 21:50:21.551892 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.551878    2564 manager.go:349] Starting recovery of all containers
Sep 18 21:50:21.562879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.562863    2564 manager.go:354] Recovery completed
Sep 18 21:50:21.563092 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.563066    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 21:50:21.563393 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.563365    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 21:50:21.567418 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.567405    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:21.569081 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.569062    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:21.569139 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.569097    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:21.569139 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.569112    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:21.569765 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.569750    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 21:50:21.569765 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.569764    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 21:50:21.569888 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.569788    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:21.569888 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.569811    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 21:50:21.570510 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.570495    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:21.570562 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.570522    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:21.570562 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.570536    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:21.573386 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.573315    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18d688b704e85dc2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-09-18 21:50:21.56908077 +0000 UTC m=+0.425736418,LastTimestamp:2026-09-18 21:50:21.56908077 +0000 UTC m=+0.425736418,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}"
Sep 18 21:50:21.590182 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.590125    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18d688b704e8bdd4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-79.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-09-18 21:50:21.569105364 +0000 UTC m=+0.425761011,LastTimestamp:2026-09-18 21:50:21.569105364 +0000 UTC m=+0.425761011,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}"
Sep 18 21:50:21.597956 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.597938    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vlpjl"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.605703    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vlpjl"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.611567    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.612694    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.612742    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.612765    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.612806    2564 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.614502    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18d688b704e8f020  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-79.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-09-18 21:50:21.56911824 +0000 UTC m=+0.425773884,LastTimestamp:2026-09-18 21:50:21.56911824 +0000 UTC m=+0.425773884,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.630368    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.630420    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-79.ec2.internal.18d688b704e85dc2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18d688b704e85dc2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-09-18 21:50:21.56908077 +0000 UTC m=+0.425736418,LastTimestamp:2026-09-18 21:50:21.570508818 +0000 UTC m=+0.427164466,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.642045    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-79.ec2.internal.18d688b704e8bdd4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18d688b704e8bdd4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-79.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-09-18 21:50:21.569105364 +0000 UTC m=+0.425761011,LastTimestamp:2026-09-18 21:50:21.570528841 +0000 UTC m=+0.427184485,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.650169    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.655335    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-79.ec2.internal.18d688b704e8f020\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18d688b704e8f020  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-79.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-09-18 21:50:21.56911824 +0000 UTC m=+0.425773884,LastTimestamp:2026-09-18 21:50:21.570541162 +0000 UTC m=+0.427196807,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.706933    2564 policy_none.go:49] "None policy: Start"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.706952    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 21:50:21.707127 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.706966    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 21:50:21.709119 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.709106    2564 policy_none.go:47] "Start"
Sep 18 21:50:21.713305 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.713241    2564 kubelet.go:2472] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Sep 18 21:50:21.746390 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.746376    2564 manager.go:337] "Starting Device Plugin manager"
Sep 18 21:50:21.746469 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.746408    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 21:50:21.746469 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.746417    2564 server.go:87] "Starting device plugin registration server"
Sep 18 21:50:21.746693 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.746681    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 21:50:21.746742 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.746697    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 21:50:21.746802 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.746787    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 21:50:21.746885 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.746872    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 21:50:21.746885 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.746885    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 21:50:21.747341 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.747318    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 21:50:21.747453 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.747359    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:21.773878 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.773859    2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-79.ec2.internal\" not found" node="ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.847049 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.847033    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:21.847827 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.847814    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:21.847902 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.847840    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:21.847902 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.847854    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:21.847902 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.847879    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.858505 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.858491    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.858567 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.858510    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-79.ec2.internal\": node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:21.876360 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.876347    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:21.914301 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.914255    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"]
Sep 18 21:50:21.914366 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.914335    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:21.915068 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.915054    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:21.915138 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.915080    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:21.915138 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.915090    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:21.917368 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.917357    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:21.917498 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.917483    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.917544 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.917520    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:21.918063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.918045    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:21.918063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.918059    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:21.918190 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.918070    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:21.918190 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.918079    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:21.918190 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.918085    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:21.918190 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.918092    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:21.920296 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.920282    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.920357 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.920307    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:21.921018 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.920999    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:21.921105 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.921030    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:21.921105 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.921043    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:21.934768 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.934746    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-79.ec2.internal\" not found" node="ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.938304 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.938289    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-79.ec2.internal\" not found" node="ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.952522 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.952504    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc3d8b9722788fbfdcb8bc7f7b68e6ec-config\") pod \"kube-apiserver-proxy-ip-10-0-130-79.ec2.internal\" (UID: \"bc3d8b9722788fbfdcb8bc7f7b68e6ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.952575 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.952525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.952575 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:21.952541    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:21.976667 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:21.976634    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.053286 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.053257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc3d8b9722788fbfdcb8bc7f7b68e6ec-config\") pod \"kube-apiserver-proxy-ip-10-0-130-79.ec2.internal\" (UID: \"bc3d8b9722788fbfdcb8bc7f7b68e6ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:22.053357 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.053340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:22.053394 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.053356    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc3d8b9722788fbfdcb8bc7f7b68e6ec-config\") pod \"kube-apiserver-proxy-ip-10-0-130-79.ec2.internal\" (UID: \"bc3d8b9722788fbfdcb8bc7f7b68e6ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:22.053394 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.053374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:22.053458 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.053340    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:22.053458 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.053404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:22.077391 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.077367    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.177925 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.177888    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.240139 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.240083    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:22.243140 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.243124    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:22.278452 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.278426    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.380147 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.380117    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.482007 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.481966    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.486488 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.486454    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 21:50:22.583357 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.583257    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.627897 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.627865    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:22.684485 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.684448    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.712454 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.712393    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 21:45:22 +0000 UTC" deadline="2028-03-18 05:12:44.183385203 +0000 UTC"
Sep 18 21:50:22.712454 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.712447    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13111h22m21.470942243s"
Sep 18 21:50:22.728545 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.728523    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:22.738389 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.738361    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:22.739606 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.739589    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:22.739696 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.739621    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:22.739696 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.739631    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:22.785172 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.785139    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.885959 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.885915    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.986520 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:22.986487    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 21:50:22.987670 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:22.987639    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f016832ad3a15792c6b1a23ee96cdaa.slice/crio-d0634e4e1aebb0f3d2ba303aa6d12de6f08d523055bedc7abe8f535871364be8 WatchSource:0}: Error finding container d0634e4e1aebb0f3d2ba303aa6d12de6f08d523055bedc7abe8f535871364be8: Status 404 returned error can't find the container with id d0634e4e1aebb0f3d2ba303aa6d12de6f08d523055bedc7abe8f535871364be8
Sep 18 21:50:22.989410 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:22.989387    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3d8b9722788fbfdcb8bc7f7b68e6ec.slice/crio-db6ebf3a16713f324305505dc69299c9c23e5b9b85b0f16808dc83ab5df24ba3 WatchSource:0}: Error finding container db6ebf3a16713f324305505dc69299c9c23e5b9b85b0f16808dc83ab5df24ba3: Status 404 returned error can't find the container with id db6ebf3a16713f324305505dc69299c9c23e5b9b85b0f16808dc83ab5df24ba3
Sep 18 21:50:22.993861 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.993845    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:50:22.996437 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:22.996417    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:23.051362 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.051321    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:23.064518 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.064495    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 21:50:23.068394 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.068379    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 21:50:23.081889 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.081870    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 21:50:23.132764 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.132685    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:23.526085 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.526000    2564 apiserver.go:52] "Watching apiserver"
Sep 18 21:50:23.536386 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.536356    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 21:50:23.536874 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.536846    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-vqfxz","openshift-network-diagnostics/network-check-target-4nh6z","openshift-network-operator/iptables-alerter-gzrln","openshift-ovn-kubernetes/ovnkube-node-9fjdd","kube-system/global-pull-secret-syncer-5nrdf","openshift-dns/node-resolver-lq478","openshift-image-registry/node-ca-qs2dt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal","openshift-multus/multus-2979h","openshift-multus/multus-additional-cni-plugins-bgqzs","kube-system/konnectivity-agent-485xb","kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj","openshift-cluster-node-tuning-operator/tuned-q6bwx"]
Sep 18 21:50:23.539339 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.539318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.542590 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.542545    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 21:50:23.542698 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.542614    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.542698 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.542626    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 21:50:23.542943 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.542924    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g6tcr\""
Sep 18 21:50:23.543215 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.543182    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 21:50:23.544836 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.544763    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.545554 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.545170    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 21:50:23.545554 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.545219    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 21:50:23.545554 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.545482    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xq7lr\""
Sep 18 21:50:23.547793 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.547458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.547793 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.547475    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 21:50:23.547793 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.547480    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:50:23.547793 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.547651    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 21:50:23.548989 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.548801    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bxpfh\""
Sep 18 21:50:23.549984 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.549963    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.550693 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.550675    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.551743    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.552037    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.552253    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.552469    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.552581    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.552758    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.552887    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5h7cd\""
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.552954    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.553076    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 21:50:23.553248 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.553254    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lp4hd\""
Sep 18 21:50:23.553825 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.553586    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.556530 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.556503    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.557629 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.557609    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 21:50:23.557730 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.557689    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4g64n\""
Sep 18 21:50:23.557796 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.557618    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 21:50:23.557865 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.557856    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 21:50:23.559689 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.559665    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:23.559797 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.559765    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:23.559933 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.559914    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4m67k\""
Sep 18 21:50:23.559991 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.559666    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 21:50:23.561736 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.561715    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 21:50:23.562053 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.562036    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 21:50:23.562361 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.562344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:23.562480 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.562405    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:23.562629 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.562604    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 21:50:23.563388 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-sysctl-d\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.563469 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563402    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-slash\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.563469 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-run-ovn\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.563469 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-node-log\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.563593 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563475    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-kubernetes\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.563593 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-sys\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.563593 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64b9919d-7825-470d-b9f7-08594294892e-tmp\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.563593 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6x6\" (UniqueName: \"kubernetes.io/projected/64b9919d-7825-470d-b9f7-08594294892e-kube-api-access-fk6x6\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.563593 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563606    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be981ec5-26f8-4362-8377-90002db40a57-host\") pod \"node-ca-qs2dt\" (UID: \"be981ec5-26f8-4362-8377-90002db40a57\") " pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/be981ec5-26f8-4362-8377-90002db40a57-serviceca\") pod \"node-ca-qs2dt\" (UID: \"be981ec5-26f8-4362-8377-90002db40a57\") " pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-systemd-units\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-var-lib-openvswitch\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-sysconfig\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563735    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-sysctl-conf\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-lib-modules\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-registration-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563812    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-sys-fs\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.563837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24mr\" (UniqueName: \"kubernetes.io/projected/0bb99213-95f5-4686-81bf-634057f76af3-kube-api-access-c24mr\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563864    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ptx\" (UniqueName: \"kubernetes.io/projected/be981ec5-26f8-4362-8377-90002db40a57-kube-api-access-z6ptx\") pod \"node-ca-qs2dt\" (UID: \"be981ec5-26f8-4362-8377-90002db40a57\") " pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-cni-bin\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563917    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-ovnkube-config\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-env-overrides\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.563983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-ovnkube-script-lib\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564014    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/02f31bd0-d812-4748-aca5-993015ea3fb7-iptables-alerter-script\") pod \"iptables-alerter-gzrln\" (UID: \"02f31bd0-d812-4748-aca5-993015ea3fb7\") " pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwcp\" (UniqueName: \"kubernetes.io/projected/02f31bd0-d812-4748-aca5-993015ea3fb7-kube-api-access-gzwcp\") pod \"iptables-alerter-gzrln\" (UID: \"02f31bd0-d812-4748-aca5-993015ea3fb7\") " pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-modprobe-d\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-host\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564115    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-socket-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564139    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-device-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-run-openvswitch\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564189    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc73221c-de83-4fd3-b462-8644f373ae8a-tmp-dir\") pod \"node-resolver-lq478\" (UID: \"bc73221c-de83-4fd3-b462-8644f373ae8a\") " pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564214    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlps5\" (UniqueName: \"kubernetes.io/projected/bc73221c-de83-4fd3-b462-8644f373ae8a-kube-api-access-nlps5\") pod \"node-resolver-lq478\" (UID: \"bc73221c-de83-4fd3-b462-8644f373ae8a\") " pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-kubelet\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.564413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-run-systemd\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-log-socket\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564333    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-systemd\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564403    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64b9919d-7825-470d-b9f7-08594294892e-etc-tuned\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lkh\" (UniqueName: \"kubernetes.io/projected/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-kube-api-access-p5lkh\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564480    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-run\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-var-lib-kubelet\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564567    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc73221c-de83-4fd3-b462-8644f373ae8a-hosts-file\") pod \"node-resolver-lq478\" (UID: \"bc73221c-de83-4fd3-b462-8644f373ae8a\") " pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02f31bd0-d812-4748-aca5-993015ea3fb7-host-slash\") pod \"iptables-alerter-gzrln\" (UID: \"02f31bd0-d812-4748-aca5-993015ea3fb7\") " pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-run-netns\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-etc-openvswitch\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-cni-netd\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.565199 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.564704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-ovn-node-metrics-cert\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.565977 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.565956    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.568489 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.568468    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:23.569417 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.569396    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 21:50:23.571087 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.569918    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-txkr2\""
Sep 18 21:50:23.572674 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.572194    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 21:50:23.573369 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.573162    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:23.573369 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.573254    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:23.574678 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.574656    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 21:50:23.574809 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.574791    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-999qd\""
Sep 18 21:50:23.575584 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.575562    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 21:50:23.576281 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.576262    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 21:50:23.602570 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.602522    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kp65s"
Sep 18 21:50:23.616865 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.616839    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kp65s"
Sep 18 21:50:23.619804 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.619754    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" event={"ID":"2f016832ad3a15792c6b1a23ee96cdaa","Type":"ContainerStarted","Data":"d0634e4e1aebb0f3d2ba303aa6d12de6f08d523055bedc7abe8f535871364be8"}
Sep 18 21:50:23.622445 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.622390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" event={"ID":"bc3d8b9722788fbfdcb8bc7f7b68e6ec","Type":"ContainerStarted","Data":"db6ebf3a16713f324305505dc69299c9c23e5b9b85b0f16808dc83ab5df24ba3"}
Sep 18 21:50:23.652818 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.652797    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 21:50:23.665153 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-system-cni-dir\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.665270 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665168    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-var-lib-cni-multus\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.665270 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665211    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc73221c-de83-4fd3-b462-8644f373ae8a-hosts-file\") pod \"node-resolver-lq478\" (UID: \"bc73221c-de83-4fd3-b462-8644f373ae8a\") " pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.665270 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-run-netns\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-etc-openvswitch\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-cni-netd\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-ovn-node-metrics-cert\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665352    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc73221c-de83-4fd3-b462-8644f373ae8a-hosts-file\") pod \"node-resolver-lq478\" (UID: \"bc73221c-de83-4fd3-b462-8644f373ae8a\") " pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665360    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-run-netns\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-slash\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-run-ovn\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64b9919d-7825-470d-b9f7-08594294892e-tmp\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665460    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-run-multus-certs\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665446    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-run-netns\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-etc-openvswitch\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.665528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665521    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/be981ec5-26f8-4362-8377-90002db40a57-serviceca\") pod \"node-ca-qs2dt\" (UID: \"be981ec5-26f8-4362-8377-90002db40a57\") " pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665538    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-slash\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-run-ovn\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-systemd-units\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665643    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-cni-netd\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-lib-modules\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-env-overrides\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665743    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-modprobe-d\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-systemd-units\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665857    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5456bfbd-26d0-451a-beac-e2567ac818e7-cnibin\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5456bfbd-26d0-451a-beac-e2567ac818e7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.665959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-lib-modules\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.666052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666043    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-modprobe-d\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666087    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666150    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6-agent-certs\") pod \"konnectivity-agent-485xb\" (UID: \"b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6\") " pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2f70242-dfd4-42f7-9661-c888e8ca26d4-dbus\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666193    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-var-lib-cni-bin\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5456bfbd-26d0-451a-beac-e2567ac818e7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666238    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-socket-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-run-openvswitch\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666380    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc73221c-de83-4fd3-b462-8644f373ae8a-tmp-dir\") pod \"node-resolver-lq478\" (UID: \"bc73221c-de83-4fd3-b462-8644f373ae8a\") " pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666410    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/be981ec5-26f8-4362-8377-90002db40a57-serviceca\") pod \"node-ca-qs2dt\" (UID: \"be981ec5-26f8-4362-8377-90002db40a57\") " pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-run-openvswitch\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlps5\" (UniqueName: \"kubernetes.io/projected/bc73221c-de83-4fd3-b462-8644f373ae8a-kube-api-access-nlps5\") pod \"node-resolver-lq478\" (UID: \"bc73221c-de83-4fd3-b462-8644f373ae8a\") " pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666446    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-socket-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-kubelet\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666512    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-run-systemd\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.666583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-run-systemd\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-kubelet\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-systemd\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666645    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-cni-dir\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-socket-dir-parent\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666689    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-systemd\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666703    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lkh\" (UniqueName: \"kubernetes.io/projected/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-kube-api-access-p5lkh\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc73221c-de83-4fd3-b462-8644f373ae8a-tmp-dir\") pod \"node-resolver-lq478\" (UID: \"bc73221c-de83-4fd3-b462-8644f373ae8a\") " pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-env-overrides\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666754    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-run\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-run\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666822    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-var-lib-kubelet\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666855    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-hostroot\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666880    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-daemon-config\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6v6m\" (UniqueName: \"kubernetes.io/projected/5456bfbd-26d0-451a-beac-e2567ac818e7-kube-api-access-g6v6m\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.667475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666996    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02f31bd0-d812-4748-aca5-993015ea3fb7-host-slash\") pod \"iptables-alerter-gzrln\" (UID: \"02f31bd0-d812-4748-aca5-993015ea3fb7\") " pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-sysctl-d\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6x6\" (UniqueName: \"kubernetes.io/projected/64b9919d-7825-470d-b9f7-08594294892e-kube-api-access-fk6x6\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.666940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-var-lib-kubelet\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667097    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02f31bd0-d812-4748-aca5-993015ea3fb7-host-slash\") pod \"iptables-alerter-gzrln\" (UID: \"02f31bd0-d812-4748-aca5-993015ea3fb7\") " pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667136    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-sysctl-d\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5456bfbd-26d0-451a-beac-e2567ac818e7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667176    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5456bfbd-26d0-451a-beac-e2567ac818e7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-node-log\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-kubernetes\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-sys\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-node-log\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667332    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-kubernetes\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667353    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-cnibin\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667335    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-sys\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xgp\" (UniqueName: \"kubernetes.io/projected/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-kube-api-access-b4xgp\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be981ec5-26f8-4362-8377-90002db40a57-host\") pod \"node-ca-qs2dt\" (UID: \"be981ec5-26f8-4362-8377-90002db40a57\") " pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667471    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be981ec5-26f8-4362-8377-90002db40a57-host\") pod \"node-ca-qs2dt\" (UID: \"be981ec5-26f8-4362-8377-90002db40a57\") " pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-var-lib-openvswitch\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-sysconfig\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-sysctl-conf\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667578    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6-konnectivity-ca\") pod \"konnectivity-agent-485xb\" (UID: \"b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6\") " pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-var-lib-openvswitch\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-cni-binary-copy\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667676    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-sysconfig\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667714    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-etc-sysctl-conf\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-registration-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667770    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-sys-fs\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24mr\" (UniqueName: \"kubernetes.io/projected/0bb99213-95f5-4686-81bf-634057f76af3-kube-api-access-c24mr\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667816    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-registration-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ptx\" (UniqueName: \"kubernetes.io/projected/be981ec5-26f8-4362-8377-90002db40a57-kube-api-access-z6ptx\") pod \"node-ca-qs2dt\" (UID: \"be981ec5-26f8-4362-8377-90002db40a57\") " pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-cni-bin\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.668980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667865    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-sys-fs\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-ovnkube-config\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-ovnkube-script-lib\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-host\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/02f31bd0-d812-4748-aca5-993015ea3fb7-iptables-alerter-script\") pod \"iptables-alerter-gzrln\" (UID: \"02f31bd0-d812-4748-aca5-993015ea3fb7\") " pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.667987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwcp\" (UniqueName: \"kubernetes.io/projected/02f31bd0-d812-4748-aca5-993015ea3fb7-kube-api-access-gzwcp\") pod \"iptables-alerter-gzrln\" (UID: \"02f31bd0-d812-4748-aca5-993015ea3fb7\") " pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668014    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2f70242-dfd4-42f7-9661-c888e8ca26d4-kubelet-config\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668041    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-run-k8s-cni-cncf-io\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-conf-dir\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5456bfbd-26d0-451a-beac-e2567ac818e7-system-cni-dir\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668477    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-device-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668505    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-etc-kubernetes\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-log-socket\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668576    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-ovnkube-config\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64b9919d-7825-470d-b9f7-08594294892e-etc-tuned\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668617    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-var-lib-kubelet\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.669950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668646    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64b9919d-7825-470d-b9f7-08594294892e-host\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69p5h\" (UniqueName: \"kubernetes.io/projected/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-kube-api-access-69p5h\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668806    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-cni-bin\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668862    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bb99213-95f5-4686-81bf-634057f76af3-device-dir\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668914    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-log-socket\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.668958    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.669145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/02f31bd0-d812-4748-aca5-993015ea3fb7-iptables-alerter-script\") pod \"iptables-alerter-gzrln\" (UID: \"02f31bd0-d812-4748-aca5-993015ea3fb7\") " pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.669808    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-ovnkube-script-lib\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.670769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-ovn-node-metrics-cert\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.670879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.670807    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64b9919d-7825-470d-b9f7-08594294892e-tmp\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.672254 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.672201    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64b9919d-7825-470d-b9f7-08594294892e-etc-tuned\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.682778 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.682412    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlps5\" (UniqueName: \"kubernetes.io/projected/bc73221c-de83-4fd3-b462-8644f373ae8a-kube-api-access-nlps5\") pod \"node-resolver-lq478\" (UID: \"bc73221c-de83-4fd3-b462-8644f373ae8a\") " pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.685636 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.685604    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ptx\" (UniqueName: \"kubernetes.io/projected/be981ec5-26f8-4362-8377-90002db40a57-kube-api-access-z6ptx\") pod \"node-ca-qs2dt\" (UID: \"be981ec5-26f8-4362-8377-90002db40a57\") " pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.686110 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.686081    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwcp\" (UniqueName: \"kubernetes.io/projected/02f31bd0-d812-4748-aca5-993015ea3fb7-kube-api-access-gzwcp\") pod \"iptables-alerter-gzrln\" (UID: \"02f31bd0-d812-4748-aca5-993015ea3fb7\") " pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.686764 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.686741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6x6\" (UniqueName: \"kubernetes.io/projected/64b9919d-7825-470d-b9f7-08594294892e-kube-api-access-fk6x6\") pod \"tuned-q6bwx\" (UID: \"64b9919d-7825-470d-b9f7-08594294892e\") " pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.687577 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.687559    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lkh\" (UniqueName: \"kubernetes.io/projected/8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b-kube-api-access-p5lkh\") pod \"ovnkube-node-9fjdd\" (UID: \"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.689818 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.689796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24mr\" (UniqueName: \"kubernetes.io/projected/0bb99213-95f5-4686-81bf-634057f76af3-kube-api-access-c24mr\") pod \"aws-ebs-csi-driver-node-z7qfj\" (UID: \"0bb99213-95f5-4686-81bf-634057f76af3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.769801 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.769767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-run-netns\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.769801 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.769811    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-run-multus-certs\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.770063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.769841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:23.770063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.769863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5456bfbd-26d0-451a-beac-e2567ac818e7-cnibin\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.770063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.769887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5456bfbd-26d0-451a-beac-e2567ac818e7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.770063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.769900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-run-netns\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.770063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.769921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6-agent-certs\") pod \"konnectivity-agent-485xb\" (UID: \"b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6\") " pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:23.770063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.769946    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2f70242-dfd4-42f7-9661-c888e8ca26d4-dbus\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:23.770382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2f70242-dfd4-42f7-9661-c888e8ca26d4-dbus\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:23.770382 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.770209    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:23.770382 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.770283    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs podName:c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba nodeName:}" failed. No retries permitted until 2026-09-18 21:50:24.27025784 +0000 UTC m=+3.126913475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs") pod "network-metrics-daemon-vqfxz" (UID: "c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:23.770578 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5456bfbd-26d0-451a-beac-e2567ac818e7-cnibin\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.770578 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-run-multus-certs\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.770578 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-var-lib-cni-bin\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.770738 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770593    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:23.770738 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-var-lib-cni-bin\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.770738 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5456bfbd-26d0-451a-beac-e2567ac818e7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.770738 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-cni-dir\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.770738 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-socket-dir-parent\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.770738 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-hostroot\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-daemon-config\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770774    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6v6m\" (UniqueName: \"kubernetes.io/projected/5456bfbd-26d0-451a-beac-e2567ac818e7-kube-api-access-g6v6m\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5456bfbd-26d0-451a-beac-e2567ac818e7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5456bfbd-26d0-451a-beac-e2567ac818e7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-cnibin\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770882    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-hostroot\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xgp\" (UniqueName: \"kubernetes.io/projected/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-kube-api-access-b4xgp\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5456bfbd-26d0-451a-beac-e2567ac818e7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6-konnectivity-ca\") pod \"konnectivity-agent-485xb\" (UID: \"b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6\") " pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770955    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-cni-dir\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-cni-binary-copy\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.770998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-socket-dir-parent\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2f70242-dfd4-42f7-9661-c888e8ca26d4-kubelet-config\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:23.771043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-run-k8s-cni-cncf-io\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771071    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-conf-dir\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5456bfbd-26d0-451a-beac-e2567ac818e7-system-cni-dir\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-etc-kubernetes\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-var-lib-kubelet\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771163    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5456bfbd-26d0-451a-beac-e2567ac818e7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69p5h\" (UniqueName: \"kubernetes.io/projected/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-kube-api-access-69p5h\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771183    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5456bfbd-26d0-451a-beac-e2567ac818e7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-system-cni-dir\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-var-lib-cni-multus\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771268    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-cnibin\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771300    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-etc-kubernetes\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771360    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2f70242-dfd4-42f7-9661-c888e8ca26d4-kubelet-config\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771366    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-var-lib-kubelet\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-run-k8s-cni-cncf-io\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771444    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-conf-dir\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771481    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5456bfbd-26d0-451a-beac-e2567ac818e7-system-cni-dir\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.771769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-system-cni-dir\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.772602 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.771600    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:23.772602 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-multus-daemon-config\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.772602 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771598    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-host-var-lib-cni-multus\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.772602 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.771650    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret podName:c2f70242-dfd4-42f7-9661-c888e8ca26d4 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:24.271633833 +0000 UTC m=+3.128289488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret") pod "global-pull-secret-syncer-5nrdf" (UID: "c2f70242-dfd4-42f7-9661-c888e8ca26d4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:23.772602 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5456bfbd-26d0-451a-beac-e2567ac818e7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.772602 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-cni-binary-copy\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.772602 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.771733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6-konnectivity-ca\") pod \"konnectivity-agent-485xb\" (UID: \"b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6\") " pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:23.774472 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.774427    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6-agent-certs\") pod \"konnectivity-agent-485xb\" (UID: \"b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6\") " pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:23.784752 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.784600    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:23.784752 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.784628    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:23.784752 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.784643    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dmpp8 for pod openshift-network-diagnostics/network-check-target-4nh6z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:23.784752 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:23.784713    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8 podName:4f08105e-3088-451f-a5ae-5c046b7ae193 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:24.28469484 +0000 UTC m=+3.141350496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dmpp8" (UniqueName: "kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8") pod "network-check-target-4nh6z" (UID: "4f08105e-3088-451f-a5ae-5c046b7ae193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:23.787829 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.787805    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6v6m\" (UniqueName: \"kubernetes.io/projected/5456bfbd-26d0-451a-beac-e2567ac818e7-kube-api-access-g6v6m\") pod \"multus-additional-cni-plugins-bgqzs\" (UID: \"5456bfbd-26d0-451a-beac-e2567ac818e7\") " pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.788052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.788027    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xgp\" (UniqueName: \"kubernetes.io/projected/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-kube-api-access-b4xgp\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:23.788777 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.788759    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69p5h\" (UniqueName: \"kubernetes.io/projected/a4f9c0dc-d777-4cc6-a84b-1563af98dd71-kube-api-access-69p5h\") pod \"multus-2979h\" (UID: \"a4f9c0dc-d777-4cc6-a84b-1563af98dd71\") " pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.852946 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.852899    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qs2dt"
Sep 18 21:50:23.864202 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.864172    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq478"
Sep 18 21:50:23.867129 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:23.867084    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe981ec5_26f8_4362_8377_90002db40a57.slice/crio-bf8df2fe8e623f650b99d9217e55f44e8a86abd7b6014f56d775c4b8c6d92b3f WatchSource:0}: Error finding container bf8df2fe8e623f650b99d9217e55f44e8a86abd7b6014f56d775c4b8c6d92b3f: Status 404 returned error can't find the container with id bf8df2fe8e623f650b99d9217e55f44e8a86abd7b6014f56d775c4b8c6d92b3f
Sep 18 21:50:23.879798 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:23.879766    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc73221c_de83_4fd3_b462_8644f373ae8a.slice/crio-22477fa25253217638a6a7cf9bc30f07e2a7ef78928fb58558511508cc366033 WatchSource:0}: Error finding container 22477fa25253217638a6a7cf9bc30f07e2a7ef78928fb58558511508cc366033: Status 404 returned error can't find the container with id 22477fa25253217638a6a7cf9bc30f07e2a7ef78928fb58558511508cc366033
Sep 18 21:50:23.881713 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.881690    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gzrln"
Sep 18 21:50:23.890936 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.890909    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q6bwx"
Sep 18 21:50:23.893452 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:23.893422    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f31bd0_d812_4748_aca5_993015ea3fb7.slice/crio-3c699b71785de6b3828d487e35f5e630ddfe74938e9534362b6b88d367cee569 WatchSource:0}: Error finding container 3c699b71785de6b3828d487e35f5e630ddfe74938e9534362b6b88d367cee569: Status 404 returned error can't find the container with id 3c699b71785de6b3828d487e35f5e630ddfe74938e9534362b6b88d367cee569
Sep 18 21:50:23.899879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.899856    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:23.901520 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:23.901497    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b9919d_7825_470d_b9f7_08594294892e.slice/crio-206b09f73375ae84e23917bdefc49a7558a98577077f1a2f63ff61c039f6b1d6 WatchSource:0}: Error finding container 206b09f73375ae84e23917bdefc49a7558a98577077f1a2f63ff61c039f6b1d6: Status 404 returned error can't find the container with id 206b09f73375ae84e23917bdefc49a7558a98577077f1a2f63ff61c039f6b1d6
Sep 18 21:50:23.909413 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.909247    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj"
Sep 18 21:50:23.912416 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:23.912326    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0b7d2d_9b1d_4ace_b2ce_9967734e2a8b.slice/crio-128ce8a7fac8edb8b7dba8e63000a12c83a83e8c230e3254bebc8ac76e4acab3 WatchSource:0}: Error finding container 128ce8a7fac8edb8b7dba8e63000a12c83a83e8c230e3254bebc8ac76e4acab3: Status 404 returned error can't find the container with id 128ce8a7fac8edb8b7dba8e63000a12c83a83e8c230e3254bebc8ac76e4acab3
Sep 18 21:50:23.918783 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.918759    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2979h"
Sep 18 21:50:23.919580 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:23.919554    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb99213_95f5_4686_81bf_634057f76af3.slice/crio-f5c49b4046d6aa3738b940b6a3d2731ca640d30532f84f4a458a754d919375bd WatchSource:0}: Error finding container f5c49b4046d6aa3738b940b6a3d2731ca640d30532f84f4a458a754d919375bd: Status 404 returned error can't find the container with id f5c49b4046d6aa3738b940b6a3d2731ca640d30532f84f4a458a754d919375bd
Sep 18 21:50:23.928296 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.928164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgqzs"
Sep 18 21:50:23.935872 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:23.935849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:23.941330 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:23.941297    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5456bfbd_26d0_451a_beac_e2567ac818e7.slice/crio-1eb013803006cf8e05828c48b5a66927088d5831476338dc9743a022e733e7d8 WatchSource:0}: Error finding container 1eb013803006cf8e05828c48b5a66927088d5831476338dc9743a022e733e7d8: Status 404 returned error can't find the container with id 1eb013803006cf8e05828c48b5a66927088d5831476338dc9743a022e733e7d8
Sep 18 21:50:23.948453 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:23.948421    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d3209a_ee8e_41b5_81c8_e0d01fc4bdd6.slice/crio-57a1172e1384ab060a0cd999809438eaed5ba3c2cbc04f1b96fd103d7a7b6b95 WatchSource:0}: Error finding container 57a1172e1384ab060a0cd999809438eaed5ba3c2cbc04f1b96fd103d7a7b6b95: Status 404 returned error can't find the container with id 57a1172e1384ab060a0cd999809438eaed5ba3c2cbc04f1b96fd103d7a7b6b95
Sep 18 21:50:24.275498 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.275411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:24.275498 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.275469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:24.275743 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:24.275604    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:24.275743 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:24.275622    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:24.275743 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:24.275678    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret podName:c2f70242-dfd4-42f7-9661-c888e8ca26d4 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:25.275659186 +0000 UTC m=+4.132314819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret") pod "global-pull-secret-syncer-5nrdf" (UID: "c2f70242-dfd4-42f7-9661-c888e8ca26d4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:24.275743 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:24.275699    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs podName:c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba nodeName:}" failed. No retries permitted until 2026-09-18 21:50:25.275687678 +0000 UTC m=+4.132343310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs") pod "network-metrics-daemon-vqfxz" (UID: "c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:24.376975 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.376938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:24.377185 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:24.377086    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:24.377185 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:24.377110    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:24.377185 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:24.377123    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dmpp8 for pod openshift-network-diagnostics/network-check-target-4nh6z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:24.377380 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:24.377196    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8 podName:4f08105e-3088-451f-a5ae-5c046b7ae193 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:25.377179386 +0000 UTC m=+4.233835020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmpp8" (UniqueName: "kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8") pod "network-check-target-4nh6z" (UID: "4f08105e-3088-451f-a5ae-5c046b7ae193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:24.613510 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.613476    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:24.613958 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:24.613605    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:24.621658 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.621623    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:45:24 +0000 UTC" deadline="2028-04-04 20:01:39.831988762 +0000 UTC"
Sep 18 21:50:24.621658 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.621658    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13534h11m15.210336034s"
Sep 18 21:50:24.626608 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.626539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqzs" event={"ID":"5456bfbd-26d0-451a-beac-e2567ac818e7","Type":"ContainerStarted","Data":"1eb013803006cf8e05828c48b5a66927088d5831476338dc9743a022e733e7d8"}
Sep 18 21:50:24.628169 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.628142    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2979h" event={"ID":"a4f9c0dc-d777-4cc6-a84b-1563af98dd71","Type":"ContainerStarted","Data":"f5f1ee28355b753f7bbabbf9c439235dea61f651ef3baa5a7d74dcae04a5e8a5"}
Sep 18 21:50:24.629572 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.629543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj" event={"ID":"0bb99213-95f5-4686-81bf-634057f76af3","Type":"ContainerStarted","Data":"f5c49b4046d6aa3738b940b6a3d2731ca640d30532f84f4a458a754d919375bd"}
Sep 18 21:50:24.631119 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.631091    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" event={"ID":"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b","Type":"ContainerStarted","Data":"128ce8a7fac8edb8b7dba8e63000a12c83a83e8c230e3254bebc8ac76e4acab3"}
Sep 18 21:50:24.633069 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.633047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gzrln" event={"ID":"02f31bd0-d812-4748-aca5-993015ea3fb7","Type":"ContainerStarted","Data":"3c699b71785de6b3828d487e35f5e630ddfe74938e9534362b6b88d367cee569"}
Sep 18 21:50:24.634563 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.634534    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq478" event={"ID":"bc73221c-de83-4fd3-b462-8644f373ae8a","Type":"ContainerStarted","Data":"22477fa25253217638a6a7cf9bc30f07e2a7ef78928fb58558511508cc366033"}
Sep 18 21:50:24.635855 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.635830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-485xb" event={"ID":"b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6","Type":"ContainerStarted","Data":"57a1172e1384ab060a0cd999809438eaed5ba3c2cbc04f1b96fd103d7a7b6b95"}
Sep 18 21:50:24.637397 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.637359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q6bwx" event={"ID":"64b9919d-7825-470d-b9f7-08594294892e","Type":"ContainerStarted","Data":"206b09f73375ae84e23917bdefc49a7558a98577077f1a2f63ff61c039f6b1d6"}
Sep 18 21:50:24.638862 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:24.638835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qs2dt" event={"ID":"be981ec5-26f8-4362-8377-90002db40a57","Type":"ContainerStarted","Data":"bf8df2fe8e623f650b99d9217e55f44e8a86abd7b6014f56d775c4b8c6d92b3f"}
Sep 18 21:50:25.286779 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:25.286714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:25.286779 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:25.286785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:25.287038 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.286931    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:25.287038 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.287000    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs podName:c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba nodeName:}" failed. No retries permitted until 2026-09-18 21:50:27.286979573 +0000 UTC m=+6.143635222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs") pod "network-metrics-daemon-vqfxz" (UID: "c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:25.287472 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.287448    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:25.287596 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.287512    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret podName:c2f70242-dfd4-42f7-9661-c888e8ca26d4 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:27.287495687 +0000 UTC m=+6.144151334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret") pod "global-pull-secret-syncer-5nrdf" (UID: "c2f70242-dfd4-42f7-9661-c888e8ca26d4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:25.388282 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:25.388189    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:25.388474 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.388408    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:25.388474 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.388436    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:25.388474 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.388451    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dmpp8 for pod openshift-network-diagnostics/network-check-target-4nh6z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:25.388658 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.388520    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8 podName:4f08105e-3088-451f-a5ae-5c046b7ae193 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:27.388498355 +0000 UTC m=+6.245153994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmpp8" (UniqueName: "kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8") pod "network-check-target-4nh6z" (UID: "4f08105e-3088-451f-a5ae-5c046b7ae193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:25.626519 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:25.622131    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:25.626519 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.622294    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:25.626519 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:25.622808    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:25.626519 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:25.622933    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:25.626519 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:25.623012    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:45:24 +0000 UTC" deadline="2028-04-24 07:07:46.576960021 +0000 UTC"
Sep 18 21:50:25.626519 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:25.623052    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14001h17m20.953912007s"
Sep 18 21:50:26.614373 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:26.613810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:26.614373 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:26.613949    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:26.668134 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:26.668094    2564 generic.go:356] "Generic (PLEG): container finished" podID="2f016832ad3a15792c6b1a23ee96cdaa" containerID="3534b9af5c9c28013e1872f7358f63626862053ef52e39439f20321a04a1825e" exitCode=0
Sep 18 21:50:26.668940 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:26.668189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" event={"ID":"2f016832ad3a15792c6b1a23ee96cdaa","Type":"ContainerDied","Data":"3534b9af5c9c28013e1872f7358f63626862053ef52e39439f20321a04a1825e"}
Sep 18 21:50:26.674504 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:26.674477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" event={"ID":"bc3d8b9722788fbfdcb8bc7f7b68e6ec","Type":"ContainerStarted","Data":"bb3d495dd6fff7c9fef0bcdd38844b959583a4707b32f49cf5824de0f8a7251b"}
Sep 18 21:50:27.307034 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:27.306994    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:27.307217 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:27.307088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:27.307301 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.307240    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:27.307356 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.307324    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret podName:c2f70242-dfd4-42f7-9661-c888e8ca26d4 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:31.307290552 +0000 UTC m=+10.163946186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret") pod "global-pull-secret-syncer-5nrdf" (UID: "c2f70242-dfd4-42f7-9661-c888e8ca26d4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:27.307431 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.307402    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:27.307484 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.307443    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs podName:c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba nodeName:}" failed. No retries permitted until 2026-09-18 21:50:31.307431051 +0000 UTC m=+10.164086698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs") pod "network-metrics-daemon-vqfxz" (UID: "c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:27.408201 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:27.408134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:27.408434 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.408369    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:27.408434 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.408390    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:27.408434 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.408403    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dmpp8 for pod openshift-network-diagnostics/network-check-target-4nh6z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:27.408607 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.408465    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8 podName:4f08105e-3088-451f-a5ae-5c046b7ae193 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:31.40844426 +0000 UTC m=+10.265099895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmpp8" (UniqueName: "kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8") pod "network-check-target-4nh6z" (UID: "4f08105e-3088-451f-a5ae-5c046b7ae193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:27.614517 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:27.614436    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:27.614740 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.614577    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:27.615189 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:27.615164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:27.615319 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:27.615264    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:27.684244 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:27.684204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" event={"ID":"2f016832ad3a15792c6b1a23ee96cdaa","Type":"ContainerStarted","Data":"e7fa879d987c90142dfdaa06106217f12d38746740fcc188e0a6383e608da1fc"}
Sep 18 21:50:27.700432 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:27.700369    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" podStartSLOduration=4.7003318499999995 podStartE2EDuration="4.70033185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:26.701652446 +0000 UTC m=+5.558308103" watchObservedRunningTime="2026-09-18 21:50:27.70033185 +0000 UTC m=+6.556987508"
Sep 18 21:50:28.613068 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:28.613035    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:28.613251 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:28.613198    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:29.613393 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:29.613359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:29.613876 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:29.613495    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:29.613986 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:29.613925    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:29.614078 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:29.614050    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:30.614333 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:30.613558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:30.614333 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:30.613942    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:31.345038 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:31.344996    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:31.345243 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:31.345089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:31.345243 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.345200    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:31.345431 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.345207    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:31.345431 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.345305    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs podName:c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba nodeName:}" failed. No retries permitted until 2026-09-18 21:50:39.345265734 +0000 UTC m=+18.201921373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs") pod "network-metrics-daemon-vqfxz" (UID: "c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:31.345431 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.345326    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret podName:c2f70242-dfd4-42f7-9661-c888e8ca26d4 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:39.345312812 +0000 UTC m=+18.201968459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret") pod "global-pull-secret-syncer-5nrdf" (UID: "c2f70242-dfd4-42f7-9661-c888e8ca26d4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:31.445829 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:31.445757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:31.446020 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.445906    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:31.446020 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.445931    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:31.446020 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.445944    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dmpp8 for pod openshift-network-diagnostics/network-check-target-4nh6z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:31.446020 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.446012    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8 podName:4f08105e-3088-451f-a5ae-5c046b7ae193 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:39.445993493 +0000 UTC m=+18.302649129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmpp8" (UniqueName: "kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8") pod "network-check-target-4nh6z" (UID: "4f08105e-3088-451f-a5ae-5c046b7ae193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:31.614950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:31.614369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:31.614950 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.614484    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:31.614950 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:31.614903    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:31.615553 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:31.615004    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:32.613648 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:32.613612    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:32.613842 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:32.613772    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:33.613940 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:33.613402    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:33.613940 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:33.613540    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:33.613940 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:33.613555    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:33.613940 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:33.613693    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:34.613503 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:34.613470    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:34.613670 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:34.613580    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:35.613385 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:35.613340    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:35.613859 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:35.613452    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:35.613859 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:35.613484    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:35.613859 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:35.613532    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:36.613209 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:36.613175    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:36.613375 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:36.613327    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:37.613676 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:37.613644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:37.614094 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:37.613766    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:37.614094 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:37.613843    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:37.614094 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:37.613948    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:38.612919 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:38.612894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:38.613120 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:38.613009    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:39.412225 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:39.412188    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:39.412687 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:39.412295    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:39.412687 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.412364    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:39.412687 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.412393    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:39.412687 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.412433    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs podName:c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba nodeName:}" failed. No retries permitted until 2026-09-18 21:50:55.412410466 +0000 UTC m=+34.269066104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs") pod "network-metrics-daemon-vqfxz" (UID: "c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:39.412687 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.412452    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret podName:c2f70242-dfd4-42f7-9661-c888e8ca26d4 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:55.412442738 +0000 UTC m=+34.269098371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret") pod "global-pull-secret-syncer-5nrdf" (UID: "c2f70242-dfd4-42f7-9661-c888e8ca26d4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:39.513326 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:39.513299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:39.513500 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.513459    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:39.513500 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.513477    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:39.513500 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.513488    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dmpp8 for pod openshift-network-diagnostics/network-check-target-4nh6z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:39.513616 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.513549    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8 podName:4f08105e-3088-451f-a5ae-5c046b7ae193 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:55.513532175 +0000 UTC m=+34.370187825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmpp8" (UniqueName: "kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8") pod "network-check-target-4nh6z" (UID: "4f08105e-3088-451f-a5ae-5c046b7ae193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:39.613529 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:39.613499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:39.613706 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:39.613508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:39.613706 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.613630    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:39.613802 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:39.613702    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:40.613549 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:40.613520    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:40.613932 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:40.613645    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:41.613960 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:41.613921    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:41.614384 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:41.614000    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:41.614384 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:41.614019    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:41.614384 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:41.614103    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:42.613896 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:42.613866    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:42.614051 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:42.613980    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:43.613481 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.613317    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:43.613633 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.613319    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:43.613633 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:43.613561    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:43.613733 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:43.613656    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:43.721539 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.721506    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qs2dt" event={"ID":"be981ec5-26f8-4362-8377-90002db40a57","Type":"ContainerStarted","Data":"7e33da3a483aee5a40ca51a19da41b8607cd9b78a9480f32fa8a42b5fac28139"}
Sep 18 21:50:43.723282 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.723231    2564 generic.go:356] "Generic (PLEG): container finished" podID="5456bfbd-26d0-451a-beac-e2567ac818e7" containerID="081433e9e0b53fc84bc095f441e74bed0b884c2113919d559e73345e22e6e5bc" exitCode=0
Sep 18 21:50:43.723417 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.723297    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqzs" event={"ID":"5456bfbd-26d0-451a-beac-e2567ac818e7","Type":"ContainerDied","Data":"081433e9e0b53fc84bc095f441e74bed0b884c2113919d559e73345e22e6e5bc"}
Sep 18 21:50:43.724996 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.724960    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2979h" event={"ID":"a4f9c0dc-d777-4cc6-a84b-1563af98dd71","Type":"ContainerStarted","Data":"0a25f502d76e606b2c03c4a4d5f183a390dc0cb86b3b8a74f57f3f6e75544af9"}
Sep 18 21:50:43.726727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.726704    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj" event={"ID":"0bb99213-95f5-4686-81bf-634057f76af3","Type":"ContainerStarted","Data":"d1078c612402f0802b85752356414fb1d9c2e0ab8e7f9114b5d8f7bf35505424"}
Sep 18 21:50:43.729928 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.729903    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" event={"ID":"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b","Type":"ContainerStarted","Data":"373c137b31ad64889c71f40c46d9cd2268707c4b5a6bf1602d0b2b01540b6aba"}
Sep 18 21:50:43.730024 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.729965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" event={"ID":"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b","Type":"ContainerStarted","Data":"d854795f21474227300adf38d168d51451096005bae465412dc6629a827a8c93"}
Sep 18 21:50:43.730024 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.730004    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" event={"ID":"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b","Type":"ContainerStarted","Data":"3645018a591906cb7718d81ed1645c3e754cef6926d8caa5ef8ca02109935cd1"}
Sep 18 21:50:43.730097 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.730037    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" event={"ID":"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b","Type":"ContainerStarted","Data":"57ee4dc14638b344074924d8865c03a9183d01342100b752fb4382624c573a85"}
Sep 18 21:50:43.730097 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.730066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" event={"ID":"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b","Type":"ContainerStarted","Data":"e72471776545ea81ab4060c6067e76601281596d5014158fa3f0f634b9d6c336"}
Sep 18 21:50:43.730170 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.730101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" event={"ID":"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b","Type":"ContainerStarted","Data":"22021e1721d23fa2f69d2c363a86b5d506196daafc617e4a61b22a98ca498c55"}
Sep 18 21:50:43.731351 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.731321    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq478" event={"ID":"bc73221c-de83-4fd3-b462-8644f373ae8a","Type":"ContainerStarted","Data":"81b42fe925d16774a26b8c2d130a14185d81c1c1dbbe97b282871121ee011098"}
Sep 18 21:50:43.732812 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.732793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-485xb" event={"ID":"b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6","Type":"ContainerStarted","Data":"49a42f876d83c99cc280d6c0d8a4a843f5dd7d21757d0d20378d5428313b5b9e"}
Sep 18 21:50:43.734352 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.734331    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q6bwx" event={"ID":"64b9919d-7825-470d-b9f7-08594294892e","Type":"ContainerStarted","Data":"cc1d808d4d0e3044f19539e463ca63cb6e455590afd9d2884c460600ae04fe6f"}
Sep 18 21:50:43.737540 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.737505    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" podStartSLOduration=20.737493705 podStartE2EDuration="20.737493705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:27.701208283 +0000 UTC m=+6.557863942" watchObservedRunningTime="2026-09-18 21:50:43.737493705 +0000 UTC m=+22.594149357"
Sep 18 21:50:43.761158 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.761113    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qs2dt" podStartSLOduration=2.622331267 podStartE2EDuration="21.761098131s" totalImagesPullingTime="19.138766864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:43.739039109 +0000 UTC m=+22.595694760" watchObservedRunningTime="2026-09-18 21:50:43.761098131 +0000 UTC m=+22.617753796"
Sep 18 21:50:43.793317 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.793249    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q6bwx" podStartSLOduration=2.669600553 podStartE2EDuration="21.793231198s" totalImagesPullingTime="19.123630645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:43.793213054 +0000 UTC m=+22.649868707" watchObservedRunningTime="2026-09-18 21:50:43.793231198 +0000 UTC m=+22.649886852"
Sep 18 21:50:43.793710 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.793667    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lq478" podStartSLOduration=2.648590553 podStartE2EDuration="21.79365686s" totalImagesPullingTime="19.145066307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:43.761730376 +0000 UTC m=+22.618386027" watchObservedRunningTime="2026-09-18 21:50:43.79365686 +0000 UTC m=+22.650312563"
Sep 18 21:50:43.821816 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.821743    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2979h" podStartSLOduration=1.50815952 podStartE2EDuration="20.821725083s" totalImagesPullingTime="19.313565563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:43.821525057 +0000 UTC m=+22.678180710" watchObservedRunningTime="2026-09-18 21:50:43.821725083 +0000 UTC m=+22.678380736"
Sep 18 21:50:43.933223 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:43.933161    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-485xb" podStartSLOduration=1.873844025 podStartE2EDuration="20.933138204s" totalImagesPullingTime="19.059294179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:43.933108749 +0000 UTC m=+22.789764401" watchObservedRunningTime="2026-09-18 21:50:43.933138204 +0000 UTC m=+22.789793857"
Sep 18 21:50:44.613104 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:44.613032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:44.613234 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:44.613183    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:44.668784 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:44.668757    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 21:50:44.738834 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:44.738794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj" event={"ID":"0bb99213-95f5-4686-81bf-634057f76af3","Type":"ContainerStarted","Data":"65d7e67105b2ccc468d3afae7f3eff0acced655b1551122157177815ec2529de"}
Sep 18 21:50:44.740530 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:44.740500    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gzrln" event={"ID":"02f31bd0-d812-4748-aca5-993015ea3fb7","Type":"ContainerStarted","Data":"7d38bc9b85adbc9df626a52ff2ebeaeeeccd12e24d96d92443e19f72ab79a0c5"}
Sep 18 21:50:44.755735 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:44.755653    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T21:50:44.668778432Z","UUID":"b55e485e-5e61-47d6-88ab-390e46fd41f1","Handler":null,"Name":"","Endpoint":""}
Sep 18 21:50:44.757375 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:44.757355    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 21:50:44.757469 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:44.757389    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 21:50:45.614057 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:45.614028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:45.614324 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:45.614126    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:45.614324 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:45.614160    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:45.614324 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:45.614237    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:46.613982 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:46.613948    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:46.614418 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:46.614088    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:46.746846 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:46.746811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj" event={"ID":"0bb99213-95f5-4686-81bf-634057f76af3","Type":"ContainerStarted","Data":"5aef308309c3e537e6f321834ea0a39d08f749804be3ce9060b49febb9526351"}
Sep 18 21:50:46.750088 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:46.750065    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" event={"ID":"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b","Type":"ContainerStarted","Data":"64a7064e0a6a6b5316301e917a5779715762a51ff2657dcafa607a5a3cedb194"}
Sep 18 21:50:46.769662 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:46.769623    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gzrln" podStartSLOduration=5.657595091 podStartE2EDuration="24.769611005s" totalImagesPullingTime="19.112015914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:44.786054622 +0000 UTC m=+23.642710272" watchObservedRunningTime="2026-09-18 21:50:46.769611005 +0000 UTC m=+25.626266715"
Sep 18 21:50:46.827026 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:46.827005    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:47.547864 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:47.547829    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:47.548401 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:47.548381    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:47.568033 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:47.567984    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7qfj" podStartSLOduration=3.02201758 podStartE2EDuration="24.567966454s" totalImagesPullingTime="21.545948874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:46.769963613 +0000 UTC m=+25.626619265" watchObservedRunningTime="2026-09-18 21:50:47.567966454 +0000 UTC m=+26.424622106"
Sep 18 21:50:47.613024 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:47.613002    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:47.613147 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:47.613115    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:47.613208 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:47.613167    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:47.613321 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:47.613288    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:47.752600 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:47.752571    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-485xb"
Sep 18 21:50:48.613715 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:48.613687    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:48.613905 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:48.613816    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:49.613458 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:49.613296    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:49.613854 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:49.613339    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:49.613854 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:49.613548    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:49.613854 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:49.613594    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:49.757392 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:49.757332    2564 generic.go:356] "Generic (PLEG): container finished" podID="5456bfbd-26d0-451a-beac-e2567ac818e7" containerID="170ad8c848f50bba4267e88b8e7e6c50bd657d554c5089d3f04b7205b0bccc50" exitCode=0
Sep 18 21:50:49.757495 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:49.757415    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqzs" event={"ID":"5456bfbd-26d0-451a-beac-e2567ac818e7","Type":"ContainerDied","Data":"170ad8c848f50bba4267e88b8e7e6c50bd657d554c5089d3f04b7205b0bccc50"}
Sep 18 21:50:49.762031 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:49.762007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" event={"ID":"8a0b7d2d-9b1d-4ace-b2ce-9967734e2a8b","Type":"ContainerStarted","Data":"d7d7dd7b806538ff95d9d48479a3d2a284c1a6073f0464988a0dfd6f4e66099c"}
Sep 18 21:50:50.613881 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:50.613849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:50.614487 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:50.613985    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:50.763975 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:50.763944    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:50.764126 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:50.764027    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:50.764126 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:50.764068    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:50.779550 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:50.779467    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:50.779917 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:50.779898    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:50:50.810457 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:50.810411    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd" podStartSLOduration=9.683008296 podStartE2EDuration="28.810396507s" totalImagesPullingTime="19.127388211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:49.817790696 +0000 UTC m=+28.674446348" watchObservedRunningTime="2026-09-18 21:50:50.810396507 +0000 UTC m=+29.667052158"
Sep 18 21:50:51.154879 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:51.154817    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4nh6z"]
Sep 18 21:50:51.155024 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:51.154938    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:51.155090 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:51.155052    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:51.163622 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:51.163592    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5nrdf"]
Sep 18 21:50:51.164124 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:51.163779    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:51.164257 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:51.164225    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:51.164257 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:51.164233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vqfxz"]
Sep 18 21:50:51.164378 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:51.164360    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:51.164471 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:51.164452    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:51.767224 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:51.767188    2564 generic.go:356] "Generic (PLEG): container finished" podID="5456bfbd-26d0-451a-beac-e2567ac818e7" containerID="c1702dfc131598caff68e2b7e0c5b09857268cba5e40d6497a9d85fd90cef7b4" exitCode=0
Sep 18 21:50:51.767738 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:51.767256    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqzs" event={"ID":"5456bfbd-26d0-451a-beac-e2567ac818e7","Type":"ContainerDied","Data":"c1702dfc131598caff68e2b7e0c5b09857268cba5e40d6497a9d85fd90cef7b4"}
Sep 18 21:50:52.613465 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:52.613322    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:52.613555 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:52.613327    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:52.613614 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:52.613553    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:52.613669 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:52.613649    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:52.613712 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:52.613336    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:52.613765 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:52.613749    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:52.771514 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:52.771424    2564 generic.go:356] "Generic (PLEG): container finished" podID="5456bfbd-26d0-451a-beac-e2567ac818e7" containerID="1277d27b63c7ada0ee1cb6470dbba6c6792185f513b0f30d4c5d7278c8b5b5cd" exitCode=0
Sep 18 21:50:52.771862 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:52.771510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqzs" event={"ID":"5456bfbd-26d0-451a-beac-e2567ac818e7","Type":"ContainerDied","Data":"1277d27b63c7ada0ee1cb6470dbba6c6792185f513b0f30d4c5d7278c8b5b5cd"}
Sep 18 21:50:54.613837 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.613807    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:54.614228 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.613807    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:54.614228 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:54.613906    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nh6z" podUID="4f08105e-3088-451f-a5ae-5c046b7ae193"
Sep 18 21:50:54.614228 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:54.614022    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:50:54.614228 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.613807    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:54.614228 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:54.614103    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nrdf" podUID="c2f70242-dfd4-42f7-9661-c888e8ca26d4"
Sep 18 21:50:54.923029 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.922959    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeReady"
Sep 18 21:50:54.923177 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.923061    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 21:50:54.985445 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.985411    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d8786bcc9-7w7lh"]
Sep 18 21:50:54.993367 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.993343    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:54.995999 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.995972    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 21:50:54.996120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.996086    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xmfvd\""
Sep 18 21:50:54.996183 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.996154    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 21:50:54.996222 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:54.996159    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 21:50:55.006230 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.006205    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d8786bcc9-7w7lh"]
Sep 18 21:50:55.016798 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.016776    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 21:50:55.022398 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.022373    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6ln68"]
Sep 18 21:50:55.026699 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.026578    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6ln68"
Sep 18 21:50:55.027635 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.027616    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tgn5c"]
Sep 18 21:50:55.028958 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.028941    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 21:50:55.029150 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.029129    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vc2r9\""
Sep 18 21:50:55.029417 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.029393    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 21:50:55.029503 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.029495    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 21:50:55.030698 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.030675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.034371 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.034171    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gb4w8\""
Sep 18 21:50:55.034371 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.034284    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 21:50:55.034371 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.034297    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 21:50:55.054947 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.054926    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tgn5c"]
Sep 18 21:50:55.064412 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.064389    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6ln68"]
Sep 18 21:50:55.126529 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126494    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/070d51d0-7645-40a7-a7e8-9d1a9376b87d-config-volume\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.126667 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-installation-pull-secrets\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.126667 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-image-registry-private-configuration\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.126667 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-ca-trust-extracted\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.126804 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126720    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.126804 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-certificates\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.126804 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126785    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.126925 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlcg5\" (UniqueName: \"kubernetes.io/projected/edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d-kube-api-access-mlcg5\") pod \"ingress-canary-6ln68\" (UID: \"edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d\") " pod="openshift-ingress-canary/ingress-canary-6ln68"
Sep 18 21:50:55.126925 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126855    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcjn7\" (UniqueName: \"kubernetes.io/projected/070d51d0-7645-40a7-a7e8-9d1a9376b87d-kube-api-access-bcjn7\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.126925 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-bound-sa-token\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.126925 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126900    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-trusted-ca\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.127115 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070d51d0-7645-40a7-a7e8-9d1a9376b87d-tmp-dir\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.127115 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.126948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d-cert\") pod \"ingress-canary-6ln68\" (UID: \"edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d\") " pod="openshift-ingress-canary/ingress-canary-6ln68"
Sep 18 21:50:55.127115 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.127002    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklws\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-kube-api-access-tklws\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.228453 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228330    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklws\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-kube-api-access-tklws\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.228453 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228388    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/070d51d0-7645-40a7-a7e8-9d1a9376b87d-config-volume\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.228453 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-installation-pull-secrets\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.228453 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-image-registry-private-configuration\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-ca-trust-extracted\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228521    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-certificates\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlcg5\" (UniqueName: \"kubernetes.io/projected/edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d-kube-api-access-mlcg5\") pod \"ingress-canary-6ln68\" (UID: \"edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d\") " pod="openshift-ingress-canary/ingress-canary-6ln68"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcjn7\" (UniqueName: \"kubernetes.io/projected/070d51d0-7645-40a7-a7e8-9d1a9376b87d-kube-api-access-bcjn7\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-bound-sa-token\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-trusted-ca\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070d51d0-7645-40a7-a7e8-9d1a9376b87d-tmp-dir\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.228776 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.228776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d-cert\") pod \"ingress-canary-6ln68\" (UID: \"edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d\") " pod="openshift-ingress-canary/ingress-canary-6ln68"
Sep 18 21:50:55.229208 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.229020    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:50:55.229208 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.229059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-ca-trust-extracted\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.229208 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.229113    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls podName:070d51d0-7645-40a7-a7e8-9d1a9376b87d nodeName:}" failed. No retries permitted until 2026-09-18 21:50:55.729093258 +0000 UTC m=+34.585748903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls") pod "dns-default-tgn5c" (UID: "070d51d0-7645-40a7-a7e8-9d1a9376b87d") : secret "dns-default-metrics-tls" not found
Sep 18 21:50:55.229426 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.229387    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:50:55.229426 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.229406    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8786bcc9-7w7lh: secret "image-registry-tls" not found
Sep 18 21:50:55.229522 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.229441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-certificates\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.229522 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.229467    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls podName:ec02faa2-62b4-4c66-9ffd-6b24ae4d9259 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:55.7294505 +0000 UTC m=+34.586106130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls") pod "image-registry-7d8786bcc9-7w7lh" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259") : secret "image-registry-tls" not found
Sep 18 21:50:55.229662 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.229630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/070d51d0-7645-40a7-a7e8-9d1a9376b87d-config-volume\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.229796 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.229675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070d51d0-7645-40a7-a7e8-9d1a9376b87d-tmp-dir\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.230188 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.230166    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-trusted-ca\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.233174 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.233139    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-image-registry-private-configuration\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.233174 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.233150    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-installation-pull-secrets\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.233340 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.233146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d-cert\") pod \"ingress-canary-6ln68\" (UID: \"edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d\") " pod="openshift-ingress-canary/ingress-canary-6ln68"
Sep 18 21:50:55.251110 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.251089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklws\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-kube-api-access-tklws\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.256642 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.256618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcjn7\" (UniqueName: \"kubernetes.io/projected/070d51d0-7645-40a7-a7e8-9d1a9376b87d-kube-api-access-bcjn7\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.256976 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.256952    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-bound-sa-token\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.258389 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.258368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlcg5\" (UniqueName: \"kubernetes.io/projected/edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d-kube-api-access-mlcg5\") pod \"ingress-canary-6ln68\" (UID: \"edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d\") " pod="openshift-ingress-canary/ingress-canary-6ln68"
Sep 18 21:50:55.340000 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.339968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6ln68"
Sep 18 21:50:55.379182 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.379140    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:50:55.430826 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.430794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:55.430984 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.430850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:55.430984 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.430939    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:55.430984 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.430960    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:55.431138 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.431009    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret podName:c2f70242-dfd4-42f7-9661-c888e8ca26d4 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:27.430996019 +0000 UTC m=+66.287651662 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret") pod "global-pull-secret-syncer-5nrdf" (UID: "c2f70242-dfd4-42f7-9661-c888e8ca26d4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:55.431138 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.431027    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs podName:c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba nodeName:}" failed. No retries permitted until 2026-09-18 21:51:27.431018277 +0000 UTC m=+66.287673931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs") pod "network-metrics-daemon-vqfxz" (UID: "c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:55.509565 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.509491    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:50:55.531389 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.531362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:55.531573 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.531550    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:55.531635 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.531584    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:55.531635 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.531600    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dmpp8 for pod openshift-network-diagnostics/network-check-target-4nh6z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:55.531737 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.531662    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8 podName:4f08105e-3088-451f-a5ae-5c046b7ae193 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:27.531642811 +0000 UTC m=+66.388298455 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dmpp8" (UniqueName: "kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8") pod "network-check-target-4nh6z" (UID: "4f08105e-3088-451f-a5ae-5c046b7ae193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:55.677935 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.677733    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6ln68"]
Sep 18 21:50:55.732920 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.732889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:55.733070 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:55.732935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:55.733135 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.733074    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:50:55.733135 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.733099    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8786bcc9-7w7lh: secret "image-registry-tls" not found
Sep 18 21:50:55.733135 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.733124    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:50:55.733294 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.733163    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls podName:ec02faa2-62b4-4c66-9ffd-6b24ae4d9259 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:56.73314362 +0000 UTC m=+35.589799254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls") pod "image-registry-7d8786bcc9-7w7lh" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259") : secret "image-registry-tls" not found
Sep 18 21:50:55.733294 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:55.733187    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls podName:070d51d0-7645-40a7-a7e8-9d1a9376b87d nodeName:}" failed. No retries permitted until 2026-09-18 21:50:56.733170439 +0000 UTC m=+35.589826069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls") pod "dns-default-tgn5c" (UID: "070d51d0-7645-40a7-a7e8-9d1a9376b87d") : secret "dns-default-metrics-tls" not found
Sep 18 21:50:56.612963 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.612930    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:50:56.613144 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.612930    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:50:56.613144 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.612938    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:50:56.620962 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.620943    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 21:50:56.624733 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.624711    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 21:50:56.624840 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.624738    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-46k8l\""
Sep 18 21:50:56.624948 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.624901    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgp2x\""
Sep 18 21:50:56.625026 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.625001    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 21:50:56.625084 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.625027    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 21:50:56.739658 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.739624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:56.740061 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:56.739663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:56.740061 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:56.739799    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:50:56.740061 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:56.739818    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8786bcc9-7w7lh: secret "image-registry-tls" not found
Sep 18 21:50:56.740061 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:56.739823    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:50:56.740061 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:56.739884    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls podName:ec02faa2-62b4-4c66-9ffd-6b24ae4d9259 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:58.739864396 +0000 UTC m=+37.596520045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls") pod "image-registry-7d8786bcc9-7w7lh" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259") : secret "image-registry-tls" not found
Sep 18 21:50:56.740061 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:56.739905    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls podName:070d51d0-7645-40a7-a7e8-9d1a9376b87d nodeName:}" failed. No retries permitted until 2026-09-18 21:50:58.739895769 +0000 UTC m=+37.596551417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls") pod "dns-default-tgn5c" (UID: "070d51d0-7645-40a7-a7e8-9d1a9376b87d") : secret "dns-default-metrics-tls" not found
Sep 18 21:50:57.144163 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:50:57.144136    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd0a4cb_7f0e_4d61_85e1_72fe73a17b2d.slice/crio-7e40ced342c6e27be962325565346784fbc0f3df317ba2cb4d583d75861b1eb6 WatchSource:0}: Error finding container 7e40ced342c6e27be962325565346784fbc0f3df317ba2cb4d583d75861b1eb6: Status 404 returned error can't find the container with id 7e40ced342c6e27be962325565346784fbc0f3df317ba2cb4d583d75861b1eb6
Sep 18 21:50:57.758439 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:57.758403    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:50:57.784597 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:57.784567    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6ln68" event={"ID":"edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d","Type":"ContainerStarted","Data":"7e40ced342c6e27be962325565346784fbc0f3df317ba2cb4d583d75861b1eb6"}
Sep 18 21:50:57.787806 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:57.787782    2564 generic.go:356] "Generic (PLEG): container finished" podID="5456bfbd-26d0-451a-beac-e2567ac818e7" containerID="0608804cb66b8aaf0acf73d521dbe934d586cfc05b776333fe1c866c2575e0ff" exitCode=0
Sep 18 21:50:57.787903 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:57.787840    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqzs" event={"ID":"5456bfbd-26d0-451a-beac-e2567ac818e7","Type":"ContainerDied","Data":"0608804cb66b8aaf0acf73d521dbe934d586cfc05b776333fe1c866c2575e0ff"}
Sep 18 21:50:57.859386 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:57.859359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:50:58.753854 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:58.753805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:50:58.753854 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:58.753854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:50:58.754075 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:58.753998    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:50:58.754075 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:58.754054    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls podName:070d51d0-7645-40a7-a7e8-9d1a9376b87d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:02.754036492 +0000 UTC m=+41.610692138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls") pod "dns-default-tgn5c" (UID: "070d51d0-7645-40a7-a7e8-9d1a9376b87d") : secret "dns-default-metrics-tls" not found
Sep 18 21:50:58.754329 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:58.754300    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:50:58.754329 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:58.754328    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8786bcc9-7w7lh: secret "image-registry-tls" not found
Sep 18 21:50:58.754489 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:50:58.754379    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls podName:ec02faa2-62b4-4c66-9ffd-6b24ae4d9259 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:02.75436466 +0000 UTC m=+41.611020303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls") pod "image-registry-7d8786bcc9-7w7lh" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259") : secret "image-registry-tls" not found
Sep 18 21:50:58.794379 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:58.794352    2564 generic.go:356] "Generic (PLEG): container finished" podID="5456bfbd-26d0-451a-beac-e2567ac818e7" containerID="88a39a0685fbc83b1dd607df95876e0c9760c00d3a8f05537f7ad4d5c00fee7f" exitCode=0
Sep 18 21:50:58.794786 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:58.794420    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqzs" event={"ID":"5456bfbd-26d0-451a-beac-e2567ac818e7","Type":"ContainerDied","Data":"88a39a0685fbc83b1dd607df95876e0c9760c00d3a8f05537f7ad4d5c00fee7f"}
Sep 18 21:50:59.798328 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:59.798293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6ln68" event={"ID":"edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d","Type":"ContainerStarted","Data":"5c352949d7a646f89661d31eb84018af4c26cc8f78c2d841c7df71132e110ee5"}
Sep 18 21:50:59.801514 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:59.801490    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqzs" event={"ID":"5456bfbd-26d0-451a-beac-e2567ac818e7","Type":"ContainerStarted","Data":"0620e5d2a5f3a3a48695944c07544138e57ce094b4717db3800a14f19afdbfe3"}
Sep 18 21:50:59.814883 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:59.814839    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6ln68" podStartSLOduration=3.411696377 podStartE2EDuration="5.814825078s" totalImagesPullingTime="2.403128701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:59.814046974 +0000 UTC m=+38.670702625" watchObservedRunningTime="2026-09-18 21:50:59.814825078 +0000 UTC m=+38.671480755"
Sep 18 21:50:59.838142 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:59.838098    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bgqzs" podStartSLOduration=5.440961697 podStartE2EDuration="36.838079648s" totalImagesPullingTime="31.397117951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:59.83703925 +0000 UTC m=+38.693694902" watchObservedRunningTime="2026-09-18 21:50:59.838079648 +0000 UTC m=+38.694735302"
Sep 18 21:50:59.964466 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:50:59.964434    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:00.066103 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:00.066066    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:01.163030 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:01.162851    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:01.241190 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:01.241152    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:02.072939 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:02.072906    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:02.253313 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:02.253280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:02.403825 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:02.403736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:02.510686 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:02.510654    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:02.606250 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:02.606225    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:02.783222 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:02.783199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:51:02.783358 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:02.783231    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:51:02.783398 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:02.783357    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:51:02.783398 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:02.783378    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8786bcc9-7w7lh: secret "image-registry-tls" not found
Sep 18 21:51:02.783473 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:02.783426    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls podName:ec02faa2-62b4-4c66-9ffd-6b24ae4d9259 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:10.783410658 +0000 UTC m=+49.640066294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls") pod "image-registry-7d8786bcc9-7w7lh" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259") : secret "image-registry-tls" not found
Sep 18 21:51:02.783473 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:02.783358    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:02.783548 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:02.783484    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls podName:070d51d0-7645-40a7-a7e8-9d1a9376b87d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:10.783469324 +0000 UTC m=+49.640124955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls") pod "dns-default-tgn5c" (UID: "070d51d0-7645-40a7-a7e8-9d1a9376b87d") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:03.106319 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:03.106235    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:03.181203 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:03.181169    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:03.526079 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:03.525994    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:03.623814 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:03.623774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:06.409280 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:06.409231    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:06.545222 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:06.545195    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:06.635401 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:06.635379    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:07.272301 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:07.272255    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:07.364036 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:07.364009    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:10.743193 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:10.743154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:10.837275 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:10.837250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:51:10.837408 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:10.837294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:51:10.837448 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:10.837409    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:10.837448 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:10.837410    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:51:10.837448 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:10.837428    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8786bcc9-7w7lh: secret "image-registry-tls" not found
Sep 18 21:51:10.837542 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:10.837463    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls podName:070d51d0-7645-40a7-a7e8-9d1a9376b87d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:26.837451204 +0000 UTC m=+65.694106834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls") pod "dns-default-tgn5c" (UID: "070d51d0-7645-40a7-a7e8-9d1a9376b87d") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:10.837542 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:10.837495    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls podName:ec02faa2-62b4-4c66-9ffd-6b24ae4d9259 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:26.837482014 +0000 UTC m=+65.694137661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls") pod "image-registry-7d8786bcc9-7w7lh" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259") : secret "image-registry-tls" not found
Sep 18 21:51:18.254093 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:18.254053    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:18.354722 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:18.354694    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:22.578844 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:22.578807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:22.670549 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:22.670520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:22.781326 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:22.781305    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fjdd"
Sep 18 21:51:23.070981 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:23.070954    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:23.311215 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:23.311182    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:23.454574 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:23.454479    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:23.760401 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:23.760320    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:23.849224 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:23.849189    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:24.138591 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:24.138561    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:24.560912 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:24.560877    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:24.949387 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:24.949293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:25.052460 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:25.052432    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:26.843644 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:26.843609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:51:26.843644 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:26.843646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:51:26.844038 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:26.843738    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:26.844038 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:26.843759    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:51:26.844038 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:26.843774    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8786bcc9-7w7lh: secret "image-registry-tls" not found
Sep 18 21:51:26.844038 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:26.843804    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls podName:070d51d0-7645-40a7-a7e8-9d1a9376b87d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:58.843792034 +0000 UTC m=+97.700447677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls") pod "dns-default-tgn5c" (UID: "070d51d0-7645-40a7-a7e8-9d1a9376b87d") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:26.844038 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:26.843817    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls podName:ec02faa2-62b4-4c66-9ffd-6b24ae4d9259 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:58.843811579 +0000 UTC m=+97.700467208 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls") pod "image-registry-7d8786bcc9-7w7lh" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259") : secret "image-registry-tls" not found
Sep 18 21:51:27.447170 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.447131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:51:27.447170 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.447179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:51:27.449862 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.449844    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 21:51:27.449912 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.449844    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 21:51:27.457399 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:27.457378    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 21:51:27.457472 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:27.457452    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs podName:c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba nodeName:}" failed. No retries permitted until 2026-09-18 21:52:31.457434093 +0000 UTC m=+130.314089740 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs") pod "network-metrics-daemon-vqfxz" (UID: "c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba") : secret "metrics-daemon-secret" not found
Sep 18 21:51:27.460287 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.460254    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f70242-dfd4-42f7-9661-c888e8ca26d4-original-pull-secret\") pod \"global-pull-secret-syncer-5nrdf\" (UID: \"c2f70242-dfd4-42f7-9661-c888e8ca26d4\") " pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:51:27.524508 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.524478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nrdf"
Sep 18 21:51:27.548598 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.548570    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:51:27.551770 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.551744    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 21:51:27.562026 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.561482    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 21:51:27.572586 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.572553    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmpp8\" (UniqueName: \"kubernetes.io/projected/4f08105e-3088-451f-a5ae-5c046b7ae193-kube-api-access-dmpp8\") pod \"network-check-target-4nh6z\" (UID: \"4f08105e-3088-451f-a5ae-5c046b7ae193\") " pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:51:27.797110 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.797084    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5nrdf"]
Sep 18 21:51:27.800750 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:51:27.800726    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f70242_dfd4_42f7_9661_c888e8ca26d4.slice/crio-1b056c4a85c2a49409aee63f0e379b0e2d2f46cef5d43aae15c33225537c2e5b WatchSource:0}: Error finding container 1b056c4a85c2a49409aee63f0e379b0e2d2f46cef5d43aae15c33225537c2e5b: Status 404 returned error can't find the container with id 1b056c4a85c2a49409aee63f0e379b0e2d2f46cef5d43aae15c33225537c2e5b
Sep 18 21:51:27.840913 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.840885    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-46k8l\""
Sep 18 21:51:27.849339 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.849318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:51:27.863405 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:27.863383    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5nrdf" event={"ID":"c2f70242-dfd4-42f7-9661-c888e8ca26d4","Type":"ContainerStarted","Data":"1b056c4a85c2a49409aee63f0e379b0e2d2f46cef5d43aae15c33225537c2e5b"}
Sep 18 21:51:28.128099 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:28.128039    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4nh6z"]
Sep 18 21:51:28.130605 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:51:28.130578    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f08105e_3088_451f_a5ae_5c046b7ae193.slice/crio-91a24bd3ca676da7a668d090d5527f9dc542664e78605d92472bb24a451470ea WatchSource:0}: Error finding container 91a24bd3ca676da7a668d090d5527f9dc542664e78605d92472bb24a451470ea: Status 404 returned error can't find the container with id 91a24bd3ca676da7a668d090d5527f9dc542664e78605d92472bb24a451470ea
Sep 18 21:51:28.867340 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:28.867304    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4nh6z" event={"ID":"4f08105e-3088-451f-a5ae-5c046b7ae193","Type":"ContainerStarted","Data":"91a24bd3ca676da7a668d090d5527f9dc542664e78605d92472bb24a451470ea"}
Sep 18 21:51:33.882569 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:33.882526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5nrdf" event={"ID":"c2f70242-dfd4-42f7-9661-c888e8ca26d4","Type":"ContainerStarted","Data":"77e8fc20ac251b05577c72ed6abad3ba5b9b3a6a4f2223c1d24d1bd26c009454"}
Sep 18 21:51:33.884070 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:33.884042    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4nh6z" event={"ID":"4f08105e-3088-451f-a5ae-5c046b7ae193","Type":"ContainerStarted","Data":"c473704f4975eb18fa9986690e8065ad392bbc3f730968981093864e38400099"}
Sep 18 21:51:33.884181 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:33.884090    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:51:33.906302 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:33.904356    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5nrdf" podStartSLOduration=65.698869409 podStartE2EDuration="1m10.904338673s" totalImagesPullingTime="5.205469264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:33.901438979 +0000 UTC m=+72.758094641" watchObservedRunningTime="2026-09-18 21:51:33.904338673 +0000 UTC m=+72.760994326"
Sep 18 21:51:33.923255 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:33.923213    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4nh6z" podStartSLOduration=66.059021998 podStartE2EDuration="1m10.923200032s" totalImagesPullingTime="4.864178034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:33.92089385 +0000 UTC m=+72.777549502" watchObservedRunningTime="2026-09-18 21:51:33.923200032 +0000 UTC m=+72.779855666"
Sep 18 21:51:36.553543 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:36.553496    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:38.443521 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:38.443485    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:39.220859 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:39.220826    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:39.289505 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:39.289477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:40.524592 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:40.524552    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:40.588129 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:40.588096    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:44.349326 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:44.349177    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:44.411997 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:44.411960    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:54.907097 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:54.907060    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:51:58.871223 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:58.871190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:51:58.871223 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:51:58.871232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:51:58.871717 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:58.871355    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:51:58.871717 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:58.871374    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8786bcc9-7w7lh: secret "image-registry-tls" not found
Sep 18 21:51:58.871717 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:58.871422    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls podName:ec02faa2-62b4-4c66-9ffd-6b24ae4d9259 nodeName:}" failed. No retries permitted until 2026-09-18 21:53:02.871407556 +0000 UTC m=+161.728063199 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls") pod "image-registry-7d8786bcc9-7w7lh" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259") : secret "image-registry-tls" not found
Sep 18 21:51:58.871717 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:58.871362    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:58.871717 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:51:58.871499    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls podName:070d51d0-7645-40a7-a7e8-9d1a9376b87d nodeName:}" failed. No retries permitted until 2026-09-18 21:53:02.871483189 +0000 UTC m=+161.728138832 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls") pod "dns-default-tgn5c" (UID: "070d51d0-7645-40a7-a7e8-9d1a9376b87d") : secret "dns-default-metrics-tls" not found
Sep 18 21:52:01.223537 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:01.223471    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:01.287562 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:01.287534    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:01.362208 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:01.362181    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:04.889022 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:04.888992    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4nh6z"
Sep 18 21:52:18.085747 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.085713    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-ws6nf"]
Sep 18 21:52:18.088434 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.088419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.090764 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.090741    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 21:52:18.091003 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.090985    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 21:52:18.091709 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.091693    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 21:52:18.092234 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.092218    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zmchf\""
Sep 18 21:52:18.092325 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.092284    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 21:52:18.097644 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.097620    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 21:52:18.098415 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.098393    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-ws6nf"]
Sep 18 21:52:18.184403 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.184379    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"]
Sep 18 21:52:18.187035 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.187021    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.189366 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.189346    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 21:52:18.189471 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.189454    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 21:52:18.189527 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.189485    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pfrsl\""
Sep 18 21:52:18.190116 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.190102    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 21:52:18.190274 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.190245    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 21:52:18.194635 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.194619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nld8l\" (UniqueName: \"kubernetes.io/projected/211fed90-33df-4a21-af33-482fd0873638-kube-api-access-nld8l\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.194718 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.194651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/211fed90-33df-4a21-af33-482fd0873638-snapshots\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.194718 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.194677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/211fed90-33df-4a21-af33-482fd0873638-serving-cert\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.194789 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.194736    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/211fed90-33df-4a21-af33-482fd0873638-tmp\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.194789 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.194763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211fed90-33df-4a21-af33-482fd0873638-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.194849 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.194793    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211fed90-33df-4a21-af33-482fd0873638-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.200834 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.200814    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"]
Sep 18 21:52:18.295895 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.295875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nld8l\" (UniqueName: \"kubernetes.io/projected/211fed90-33df-4a21-af33-482fd0873638-kube-api-access-nld8l\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.296009 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.295907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b46m8\" (UniqueName: \"kubernetes.io/projected/8e61b030-2e75-4215-9cd0-db32078129c1-kube-api-access-b46m8\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.296009 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.295933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/211fed90-33df-4a21-af33-482fd0873638-snapshots\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.296009 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.295963    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/211fed90-33df-4a21-af33-482fd0873638-serving-cert\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.296009 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.295988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8e61b030-2e75-4215-9cd0-db32078129c1-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.296009 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.296008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/211fed90-33df-4a21-af33-482fd0873638-tmp\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.296232 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.296033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211fed90-33df-4a21-af33-482fd0873638-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.296232 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.296059    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.296232 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.296080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211fed90-33df-4a21-af33-482fd0873638-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.296675 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.296651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/211fed90-33df-4a21-af33-482fd0873638-snapshots\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.296722 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.296651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/211fed90-33df-4a21-af33-482fd0873638-tmp\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.296800 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.296779    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211fed90-33df-4a21-af33-482fd0873638-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.297196 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.297181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211fed90-33df-4a21-af33-482fd0873638-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.298198 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.298181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/211fed90-33df-4a21-af33-482fd0873638-serving-cert\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.304364 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.304345    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nld8l\" (UniqueName: \"kubernetes.io/projected/211fed90-33df-4a21-af33-482fd0873638-kube-api-access-nld8l\") pod \"insights-operator-6f9c644dd9-ws6nf\" (UID: \"211fed90-33df-4a21-af33-482fd0873638\") " pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.396395 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.396348    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8e61b030-2e75-4215-9cd0-db32078129c1-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.396395 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.396384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.396517 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.396419    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b46m8\" (UniqueName: \"kubernetes.io/projected/8e61b030-2e75-4215-9cd0-db32078129c1-kube-api-access-b46m8\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.396517 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:18.396498    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:18.396592 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:18.396547    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls podName:8e61b030-2e75-4215-9cd0-db32078129c1 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:18.896533218 +0000 UTC m=+117.753188864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pmrll" (UID: "8e61b030-2e75-4215-9cd0-db32078129c1") : secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:18.397050 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.397031    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8e61b030-2e75-4215-9cd0-db32078129c1-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.398366 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.398353    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf"
Sep 18 21:52:18.410013 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.409993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b46m8\" (UniqueName: \"kubernetes.io/projected/8e61b030-2e75-4215-9cd0-db32078129c1-kube-api-access-b46m8\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.667285 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.667202    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-ws6nf"]
Sep 18 21:52:18.670142 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:52:18.670119    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211fed90_33df_4a21_af33_482fd0873638.slice/crio-644a640b2d08a6e46ffe4cdec336d898a4c9aea37111b6234e2034001e4d7fff WatchSource:0}: Error finding container 644a640b2d08a6e46ffe4cdec336d898a4c9aea37111b6234e2034001e4d7fff: Status 404 returned error can't find the container with id 644a640b2d08a6e46ffe4cdec336d898a4c9aea37111b6234e2034001e4d7fff
Sep 18 21:52:18.899216 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.899183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:18.899408 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:18.899311    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:18.899408 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:18.899366    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls podName:8e61b030-2e75-4215-9cd0-db32078129c1 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:19.899352595 +0000 UTC m=+118.756008239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pmrll" (UID: "8e61b030-2e75-4215-9cd0-db32078129c1") : secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:18.987900 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:18.987828    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf" event={"ID":"211fed90-33df-4a21-af33-482fd0873638","Type":"ContainerStarted","Data":"644a640b2d08a6e46ffe4cdec336d898a4c9aea37111b6234e2034001e4d7fff"}
Sep 18 21:52:19.907102 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:19.907069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:19.907466 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:19.907241    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:19.907466 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:19.907345    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls podName:8e61b030-2e75-4215-9cd0-db32078129c1 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:21.907323024 +0000 UTC m=+120.763978677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pmrll" (UID: "8e61b030-2e75-4215-9cd0-db32078129c1") : secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:20.994086 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:20.994050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf" event={"ID":"211fed90-33df-4a21-af33-482fd0873638","Type":"ContainerStarted","Data":"728f9a7a279e5e5a028b5ed4afc0a8b1c1c33fc9afb98e337a9faf337a29dfd0"}
Sep 18 21:52:21.371655 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.371594    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf" podStartSLOduration=1.622854848 podStartE2EDuration="3.371571486s" totalImagesPullingTime="1.748716638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:21.013651854 +0000 UTC m=+119.870307506" watchObservedRunningTime="2026-09-18 21:52:21.371571486 +0000 UTC m=+120.228227138"
Sep 18 21:52:21.372600 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.372561    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6jk4g"]
Sep 18 21:52:21.372873 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.372845    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:21.376030 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.376015    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.383338 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:21.383313    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-6jk4g\" is forbidden: User \"system:node:ip-10-0-130-79.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-79.ec2.internal' and this object" podUID="933fc10b-a510-420e-8cae-3a99f8061cd5" pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.383418 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:21.383335    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-130-79.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-79.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Sep 18 21:52:21.383418 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:21.383335    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-pmpgg\" is forbidden: User \"system:node:ip-10-0-130-79.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-79.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pmpgg\"" type="*v1.Secret"
Sep 18 21:52:21.383418 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:21.383340    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-130-79.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-79.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap"
Sep 18 21:52:21.394905 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.394884    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6jk4g"]
Sep 18 21:52:21.508318 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.508293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:21.519232 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.519208    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/933fc10b-a510-420e-8cae-3a99f8061cd5-data-volume\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.519327 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.519282    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmd2\" (UniqueName: \"kubernetes.io/projected/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-api-access-wmmd2\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.519408 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.519339    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.519408 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.519379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.519516 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.519451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/933fc10b-a510-420e-8cae-3a99f8061cd5-crio-socket\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.619786 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.619766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.619907 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.619809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.619907 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.619844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/933fc10b-a510-420e-8cae-3a99f8061cd5-crio-socket\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.619907 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.619883    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/933fc10b-a510-420e-8cae-3a99f8061cd5-data-volume\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.620063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.619928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmd2\" (UniqueName: \"kubernetes.io/projected/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-api-access-wmmd2\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.620063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.619948    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/933fc10b-a510-420e-8cae-3a99f8061cd5-crio-socket\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.620230 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.620211    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/933fc10b-a510-420e-8cae-3a99f8061cd5-data-volume\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.630842 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.630788    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmd2\" (UniqueName: \"kubernetes.io/projected/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-api-access-wmmd2\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:21.922389 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:21.922319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:21.922518 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:21.922433    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:21.922518 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:21.922484    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls podName:8e61b030-2e75-4215-9cd0-db32078129c1 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:25.922471053 +0000 UTC m=+124.779126697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pmrll" (UID: "8e61b030-2e75-4215-9cd0-db32078129c1") : secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:22.393255 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:22.393215    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 21:52:22.400464 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:22.400437    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:22.400591 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:22.400503    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:22.900486693 +0000 UTC m=+121.757142334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:22.439632 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:22.439609    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 21:52:22.440662 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:22.440646    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:22.440737 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:22.440713    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:22.940692827 +0000 UTC m=+121.797348471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:22.734062 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:22.733989    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pmpgg\""
Sep 18 21:52:22.928947 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:22.928919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:22.929086 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:22.929031    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:22.929124 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:22.929090    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:23.929076606 +0000 UTC m=+122.785732236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:23.029391 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:23.029368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:23.029498 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:23.029480    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:23.029546 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:23.029526    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:24.029512163 +0000 UTC m=+122.886167806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:23.277549 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:23.277526    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lq478_bc73221c-de83-4fd3-b462-8644f373ae8a/dns-node-resolver/0.log"
Sep 18 21:52:23.935339 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:23.935301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:23.935720 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:23.935427    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:23.935720 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:23.935506    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:25.935488853 +0000 UTC m=+124.792144508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:24.036663 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:24.036618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:24.036846 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:24.036764    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:24.036846 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:24.036827    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:26.03681303 +0000 UTC m=+124.893468660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:24.077571 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:24.077546    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qs2dt_be981ec5-26f8-4362-8377-90002db40a57/node-ca/0.log"
Sep 18 21:52:24.485131 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:24.485102    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6ln68_edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d/serve-healthcheck-canary/0.log"
Sep 18 21:52:25.953691 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:25.953655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:25.954145 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:25.953710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:25.954145 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:25.953797    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:25.954145 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:25.953815    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:25.954145 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:25.953868    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls podName:8e61b030-2e75-4215-9cd0-db32078129c1 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:33.953855308 +0000 UTC m=+132.810510937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pmrll" (UID: "8e61b030-2e75-4215-9cd0-db32078129c1") : secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:25.954145 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:25.953880    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:29.95387403 +0000 UTC m=+128.810529659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:26.055133 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:26.055088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:26.055321 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:26.055214    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:26.055321 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:26.055292    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:30.055256361 +0000 UTC m=+128.911912009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:28.435215 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:28.435178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:28.536638 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:28.536609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:29.986903 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:29.986860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:29.987305 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:29.986947    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:29.987305 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:29.987010    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:37.986995363 +0000 UTC m=+136.843651006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:30.087261 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:30.087228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:30.087425 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:30.087374    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:30.087468 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:30.087449    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:38.08743322 +0000 UTC m=+136.944088884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:31.497304 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:31.497247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:52:31.497739 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:31.497389    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 21:52:31.497739 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:31.497463    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs podName:c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba nodeName:}" failed. No retries permitted until 2026-09-18 21:54:33.49744636 +0000 UTC m=+252.354101990 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs") pod "network-metrics-daemon-vqfxz" (UID: "c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba") : secret "metrics-daemon-secret" not found
Sep 18 21:52:31.561786 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:31.561755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:31.630196 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:31.630167    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:32.780183 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:32.780150    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:32.848426 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:32.848400    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:33.227409 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:33.227338    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:33.294484 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:33.294458    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:33.570041 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:33.570014    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:33.662125 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:33.662095    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:34.017630 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.017549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:34.017969 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:34.017692    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:34.017969 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:34.017754    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls podName:8e61b030-2e75-4215-9cd0-db32078129c1 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:50.01774069 +0000 UTC m=+148.874396334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pmrll" (UID: "8e61b030-2e75-4215-9cd0-db32078129c1") : secret "cluster-monitoring-operator-tls" not found
Sep 18 21:52:34.527214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.527183    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:34.540584 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.540560    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw"]
Sep 18 21:52:34.544659 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.544645    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw"
Sep 18 21:52:34.547228 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.547206    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 21:52:34.551046 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.551026    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 21:52:34.551046 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.551036    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ghw76\""
Sep 18 21:52:34.556515 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.556487    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw"]
Sep 18 21:52:34.622049 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.622023    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:34.684156 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.684128    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:34.722411 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.722388    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hgl\" (UniqueName: \"kubernetes.io/projected/126f7e2c-cb3f-4301-9820-305e42c92369-kube-api-access-26hgl\") pod \"migrator-76794cbd6c-kq9nw\" (UID: \"126f7e2c-cb3f-4301-9820-305e42c92369\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw"
Sep 18 21:52:34.823015 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.822933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hgl\" (UniqueName: \"kubernetes.io/projected/126f7e2c-cb3f-4301-9820-305e42c92369-kube-api-access-26hgl\") pod \"migrator-76794cbd6c-kq9nw\" (UID: \"126f7e2c-cb3f-4301-9820-305e42c92369\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw"
Sep 18 21:52:34.834123 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.834096    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hgl\" (UniqueName: \"kubernetes.io/projected/126f7e2c-cb3f-4301-9820-305e42c92369-kube-api-access-26hgl\") pod \"migrator-76794cbd6c-kq9nw\" (UID: \"126f7e2c-cb3f-4301-9820-305e42c92369\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw"
Sep 18 21:52:34.856108 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:34.856088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw"
Sep 18 21:52:35.153547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.153524    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw"]
Sep 18 21:52:35.155688 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.155664    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pmkrf"]
Sep 18 21:52:35.155820 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:52:35.155798    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126f7e2c_cb3f_4301_9820_305e42c92369.slice/crio-acc2345e26e137a15a24c546f118b9ab2a8eac8c9141f9a6252573916a5bb316 WatchSource:0}: Error finding container acc2345e26e137a15a24c546f118b9ab2a8eac8c9141f9a6252573916a5bb316: Status 404 returned error can't find the container with id acc2345e26e137a15a24c546f118b9ab2a8eac8c9141f9a6252573916a5bb316
Sep 18 21:52:35.159824 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.159749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.164000 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.163981    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vm47b\""
Sep 18 21:52:35.164951 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.164932    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 21:52:35.166384 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.166367    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 21:52:35.166467 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.166402    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 21:52:35.173192 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.173170    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 21:52:35.183044 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.183026    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pmkrf"]
Sep 18 21:52:35.211494 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.211469    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:35.312754 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.312726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:35.326724 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.326699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmzdg\" (UniqueName: \"kubernetes.io/projected/8e95e996-62bd-4eb8-a737-5245be5e805d-kube-api-access-kmzdg\") pod \"service-ca-55f6fc7b86-pmkrf\" (UID: \"8e95e996-62bd-4eb8-a737-5245be5e805d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.326810 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.326763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8e95e996-62bd-4eb8-a737-5245be5e805d-signing-key\") pod \"service-ca-55f6fc7b86-pmkrf\" (UID: \"8e95e996-62bd-4eb8-a737-5245be5e805d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.326851 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.326836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8e95e996-62bd-4eb8-a737-5245be5e805d-signing-cabundle\") pod \"service-ca-55f6fc7b86-pmkrf\" (UID: \"8e95e996-62bd-4eb8-a737-5245be5e805d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.428181 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.428111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmzdg\" (UniqueName: \"kubernetes.io/projected/8e95e996-62bd-4eb8-a737-5245be5e805d-kube-api-access-kmzdg\") pod \"service-ca-55f6fc7b86-pmkrf\" (UID: \"8e95e996-62bd-4eb8-a737-5245be5e805d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.428181 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.428178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8e95e996-62bd-4eb8-a737-5245be5e805d-signing-key\") pod \"service-ca-55f6fc7b86-pmkrf\" (UID: \"8e95e996-62bd-4eb8-a737-5245be5e805d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.428356 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.428223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8e95e996-62bd-4eb8-a737-5245be5e805d-signing-cabundle\") pod \"service-ca-55f6fc7b86-pmkrf\" (UID: \"8e95e996-62bd-4eb8-a737-5245be5e805d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.428866 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.428832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8e95e996-62bd-4eb8-a737-5245be5e805d-signing-cabundle\") pod \"service-ca-55f6fc7b86-pmkrf\" (UID: \"8e95e996-62bd-4eb8-a737-5245be5e805d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.430507 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.430486    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8e95e996-62bd-4eb8-a737-5245be5e805d-signing-key\") pod \"service-ca-55f6fc7b86-pmkrf\" (UID: \"8e95e996-62bd-4eb8-a737-5245be5e805d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.437393 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.437372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmzdg\" (UniqueName: \"kubernetes.io/projected/8e95e996-62bd-4eb8-a737-5245be5e805d-kube-api-access-kmzdg\") pod \"service-ca-55f6fc7b86-pmkrf\" (UID: \"8e95e996-62bd-4eb8-a737-5245be5e805d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.468327 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.468307    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf"
Sep 18 21:52:35.743038 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:35.742969    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pmkrf"]
Sep 18 21:52:35.746207 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:52:35.746182    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e95e996_62bd_4eb8_a737_5245be5e805d.slice/crio-242f349951d4c039ae58011e724ce5877bde9839e2f08df44907a84e0942c494 WatchSource:0}: Error finding container 242f349951d4c039ae58011e724ce5877bde9839e2f08df44907a84e0942c494: Status 404 returned error can't find the container with id 242f349951d4c039ae58011e724ce5877bde9839e2f08df44907a84e0942c494
Sep 18 21:52:36.031681 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:36.031648    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf" event={"ID":"8e95e996-62bd-4eb8-a737-5245be5e805d","Type":"ContainerStarted","Data":"242f349951d4c039ae58011e724ce5877bde9839e2f08df44907a84e0942c494"}
Sep 18 21:52:36.032952 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:36.032920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw" event={"ID":"126f7e2c-cb3f-4301-9820-305e42c92369","Type":"ContainerStarted","Data":"acc2345e26e137a15a24c546f118b9ab2a8eac8c9141f9a6252573916a5bb316"}
Sep 18 21:52:36.217752 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:36.217694    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:36.330051 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:36.329967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:36.431155 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:36.431118    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:37.037807 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:37.037777    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw" event={"ID":"126f7e2c-cb3f-4301-9820-305e42c92369","Type":"ContainerStarted","Data":"d1f6f4574ab2f6fa7f52d95425b569e44c15492c032e5461a61fa231fcb5eee5"}
Sep 18 21:52:37.037952 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:37.037827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw" event={"ID":"126f7e2c-cb3f-4301-9820-305e42c92369","Type":"ContainerStarted","Data":"5886f198a93f07859cb8afac3be3587b57f7b59adeb6123429b64c6cd03e5fda"}
Sep 18 21:52:37.056779 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:37.056728    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kq9nw" podStartSLOduration=1.669777845 podStartE2EDuration="3.056707986s" totalImagesPullingTime="1.386930141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:37.055299503 +0000 UTC m=+135.911955155" watchObservedRunningTime="2026-09-18 21:52:37.056707986 +0000 UTC m=+135.913363642"
Sep 18 21:52:38.042482 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:38.042440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf" event={"ID":"8e95e996-62bd-4eb8-a737-5245be5e805d","Type":"ContainerStarted","Data":"54a4e07974b2e617a19aa466e00c7dd88035979a7676f34b1a17b73300ba7e15"}
Sep 18 21:52:38.051058 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:38.051035    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:38.051500 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:38.051482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/933fc10b-a510-420e-8cae-3a99f8061cd5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:38.063220 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:38.063179    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-pmkrf" podStartSLOduration=1.180552757 podStartE2EDuration="3.063166331s" totalImagesPullingTime="1.882613574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:38.062808476 +0000 UTC m=+136.919464131" watchObservedRunningTime="2026-09-18 21:52:38.063166331 +0000 UTC m=+136.919821983"
Sep 18 21:52:38.151612 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:38.151590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:38.151718 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:38.151681    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:38.151755 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:38.151733    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls podName:933fc10b-a510-420e-8cae-3a99f8061cd5 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:54.151721073 +0000 UTC m=+153.008376717 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6jk4g" (UID: "933fc10b-a510-420e-8cae-3a99f8061cd5") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:39.267307 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:39.267258    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:39.381581 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:39.381544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:39.468705 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:39.468677    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:39.569102 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:39.569050    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:39.639497 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:39.639463    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:39.709728 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:39.709654    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:39.808668 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:39.808642    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:39.967999 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:39.967920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:40.753278 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:40.753241    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:40.951147 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:40.951113    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:41.358041 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:41.358006    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:41.760009 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:41.759923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:42.158005 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:42.157960    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:42.756840 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:42.756807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:42.960505 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:42.960469    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:43.155107 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:43.155067    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:43.236341 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:43.236309    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:43.348385 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:43.348344    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:43.449666 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:43.449585    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:43.780319 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:43.780282    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:44.364319 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:44.364283    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:44.560680 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:44.560651    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:44.961014 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:44.960977    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:45.552843 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:45.552806    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:45.653705 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:45.653669    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:45.948542 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:45.948443    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:46.362097 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:46.362065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:46.789524 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:46.789493    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:46.966009 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:46.965971    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:47.568336 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:47.568301    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:47.948851 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:47.948770    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:48.156218 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:48.156182    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:48.356538 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:48.356480    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:48.563457 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:48.563424    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:48.966205 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:48.966176    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:49.557166 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:49.557130    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:49.758343 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:49.758313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:49.947148 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:49.947059    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:50.043751 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:50.043714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:50.046129 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:50.046101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e61b030-2e75-4215-9cd0-db32078129c1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pmrll\" (UID: \"8e61b030-2e75-4215-9cd0-db32078129c1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:50.297941 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:50.297915    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pfrsl\""
Sep 18 21:52:50.306087 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:50.306060    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"
Sep 18 21:52:50.576905 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:50.576838    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll"]
Sep 18 21:52:50.579598 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:52:50.579571    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e61b030_2e75_4215_9cd0_db32078129c1.slice/crio-eb96f8ae2680a311b0422354d73b95ddd6a854d14789322207e897d87d9edf03 WatchSource:0}: Error finding container eb96f8ae2680a311b0422354d73b95ddd6a854d14789322207e897d87d9edf03: Status 404 returned error can't find the container with id eb96f8ae2680a311b0422354d73b95ddd6a854d14789322207e897d87d9edf03
Sep 18 21:52:51.079700 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:51.079658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll" event={"ID":"8e61b030-2e75-4215-9cd0-db32078129c1","Type":"ContainerStarted","Data":"eb96f8ae2680a311b0422354d73b95ddd6a854d14789322207e897d87d9edf03"}
Sep 18 21:52:51.325908 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:51.325866    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:51.399235 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:51.399162    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:52.743467 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:52.743392    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:52.898464 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:52.898423    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:53.088983 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:53.088952    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll" event={"ID":"8e61b030-2e75-4215-9cd0-db32078129c1","Type":"ContainerStarted","Data":"fd03d3f7c5c13b31dadcf0f7b66b53bba57935df03d7ae797e9efd2152c07ad9"}
Sep 18 21:52:53.108520 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:53.108475    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pmrll" podStartSLOduration=33.297161739 podStartE2EDuration="35.10845724s" totalImagesPullingTime="1.811295501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:53.107492816 +0000 UTC m=+151.964148469" watchObservedRunningTime="2026-09-18 21:52:53.10845724 +0000 UTC m=+151.965112891"
Sep 18 21:52:53.324317 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:53.324291    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:54.176507 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:54.176459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:54.178751 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:54.178734    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/933fc10b-a510-420e-8cae-3a99f8061cd5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6jk4g\" (UID: \"933fc10b-a510-420e-8cae-3a99f8061cd5\") " pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:54.385461 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:54.385436    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6jk4g"
Sep 18 21:52:54.661854 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:52:54.661819    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933fc10b_a510_420e_8cae_3a99f8061cd5.slice/crio-61b4be899c4377326217693112794cc3d47643a70b257190030f2565ad5bf2b3 WatchSource:0}: Error finding container 61b4be899c4377326217693112794cc3d47643a70b257190030f2565ad5bf2b3: Status 404 returned error can't find the container with id 61b4be899c4377326217693112794cc3d47643a70b257190030f2565ad5bf2b3
Sep 18 21:52:54.662242 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:54.662216    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6jk4g"]
Sep 18 21:52:55.096341 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.096288    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6jk4g" event={"ID":"933fc10b-a510-420e-8cae-3a99f8061cd5","Type":"ContainerStarted","Data":"9bedc9da8c3f94311332efe8d456d591e5c57db7c4633117701d36122fd03f5a"}
Sep 18 21:52:55.096493 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.096378    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6jk4g" event={"ID":"933fc10b-a510-420e-8cae-3a99f8061cd5","Type":"ContainerStarted","Data":"61b4be899c4377326217693112794cc3d47643a70b257190030f2565ad5bf2b3"}
Sep 18 21:52:55.223253 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.223219    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-v598q"]
Sep 18 21:52:55.236009 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.235982    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q"
Sep 18 21:52:55.241049 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.241026    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 21:52:55.243305 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.243284    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 21:52:55.245310 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.245291    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-c7j8h\""
Sep 18 21:52:55.249514 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.249492    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-v598q"]
Sep 18 21:52:55.285144 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.285109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f3133e4f-8bf5-41fe-8467-ca57a8a1a309-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-v598q\" (UID: \"f3133e4f-8bf5-41fe-8467-ca57a8a1a309\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q"
Sep 18 21:52:55.285339 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.285306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3133e4f-8bf5-41fe-8467-ca57a8a1a309-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-v598q\" (UID: \"f3133e4f-8bf5-41fe-8467-ca57a8a1a309\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q"
Sep 18 21:52:55.311540 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.311512    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"]
Sep 18 21:52:55.334056 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.334033    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm"]
Sep 18 21:52:55.334209 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.334173    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"
Sep 18 21:52:55.338663 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.338636    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 21:52:55.339419 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.339397    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pksb2\""
Sep 18 21:52:55.345622 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.345602    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm"]
Sep 18 21:52:55.345744 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.345639    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"]
Sep 18 21:52:55.345800 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.345743    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm"
Sep 18 21:52:55.352095 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.352080    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qp6cx\""
Sep 18 21:52:55.356775 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.356747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:55.386047 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.386028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f3133e4f-8bf5-41fe-8467-ca57a8a1a309-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-v598q\" (UID: \"f3133e4f-8bf5-41fe-8467-ca57a8a1a309\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q"
Sep 18 21:52:55.386150 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.386131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/865926b3-b843-49ce-bd12-591aa6b8c33a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-lqv8c\" (UID: \"865926b3-b843-49ce-bd12-591aa6b8c33a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"
Sep 18 21:52:55.386200 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.386189    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3133e4f-8bf5-41fe-8467-ca57a8a1a309-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-v598q\" (UID: \"f3133e4f-8bf5-41fe-8467-ca57a8a1a309\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q"
Sep 18 21:52:55.386249 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.386218    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwpj4\" (UniqueName: \"kubernetes.io/projected/b4877d2e-358b-40b0-be2a-c86b2196e7ac-kube-api-access-xwpj4\") pod \"network-check-source-7cbc45fddc-2ddwm\" (UID: \"b4877d2e-358b-40b0-be2a-c86b2196e7ac\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm"
Sep 18 21:52:55.386650 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.386632    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f3133e4f-8bf5-41fe-8467-ca57a8a1a309-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-v598q\" (UID: \"f3133e4f-8bf5-41fe-8467-ca57a8a1a309\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q"
Sep 18 21:52:55.388570 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.388551    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3133e4f-8bf5-41fe-8467-ca57a8a1a309-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-v598q\" (UID: \"f3133e4f-8bf5-41fe-8467-ca57a8a1a309\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q"
Sep 18 21:52:55.486716 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.486694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/865926b3-b843-49ce-bd12-591aa6b8c33a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-lqv8c\" (UID: \"865926b3-b843-49ce-bd12-591aa6b8c33a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"
Sep 18 21:52:55.486845 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.486751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwpj4\" (UniqueName: \"kubernetes.io/projected/b4877d2e-358b-40b0-be2a-c86b2196e7ac-kube-api-access-xwpj4\") pod \"network-check-source-7cbc45fddc-2ddwm\" (UID: \"b4877d2e-358b-40b0-be2a-c86b2196e7ac\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm"
Sep 18 21:52:55.488865 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.488846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/865926b3-b843-49ce-bd12-591aa6b8c33a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-lqv8c\" (UID: \"865926b3-b843-49ce-bd12-591aa6b8c33a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"
Sep 18 21:52:55.500428 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.500399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:55.503786 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.503761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwpj4\" (UniqueName: \"kubernetes.io/projected/b4877d2e-358b-40b0-be2a-c86b2196e7ac-kube-api-access-xwpj4\") pod \"network-check-source-7cbc45fddc-2ddwm\" (UID: \"b4877d2e-358b-40b0-be2a-c86b2196e7ac\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm"
Sep 18 21:52:55.543966 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.543945    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q"
Sep 18 21:52:55.645067 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.645000    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"
Sep 18 21:52:55.653980 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.653959    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm"
Sep 18 21:52:55.823172 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.823139    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-v598q"]
Sep 18 21:52:55.826252 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:52:55.826211    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3133e4f_8bf5_41fe_8467_ca57a8a1a309.slice/crio-f663420eb28dde0ef9f4ad0764f03ad6f92df9d61e1a5af20d8cb29888896524 WatchSource:0}: Error finding container f663420eb28dde0ef9f4ad0764f03ad6f92df9d61e1a5af20d8cb29888896524: Status 404 returned error can't find the container with id f663420eb28dde0ef9f4ad0764f03ad6f92df9d61e1a5af20d8cb29888896524
Sep 18 21:52:55.922476 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.922402    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"]
Sep 18 21:52:55.925461 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:52:55.925433    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865926b3_b843_49ce_bd12_591aa6b8c33a.slice/crio-32341394e645df5775a7c4cec1d08d1f2e16b10e6d2d4e4bed82d439cae0aab5 WatchSource:0}: Error finding container 32341394e645df5775a7c4cec1d08d1f2e16b10e6d2d4e4bed82d439cae0aab5: Status 404 returned error can't find the container with id 32341394e645df5775a7c4cec1d08d1f2e16b10e6d2d4e4bed82d439cae0aab5
Sep 18 21:52:55.941315 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:55.941293    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm"]
Sep 18 21:52:55.944828 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:52:55.944801    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4877d2e_358b_40b0_be2a_c86b2196e7ac.slice/crio-aca53948ab147cd070956fd735da8956f34edd4820732e3d1e309349f955153c WatchSource:0}: Error finding container aca53948ab147cd070956fd735da8956f34edd4820732e3d1e309349f955153c: Status 404 returned error can't find the container with id aca53948ab147cd070956fd735da8956f34edd4820732e3d1e309349f955153c
Sep 18 21:52:56.101574 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:56.101541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q" event={"ID":"f3133e4f-8bf5-41fe-8467-ca57a8a1a309","Type":"ContainerStarted","Data":"f663420eb28dde0ef9f4ad0764f03ad6f92df9d61e1a5af20d8cb29888896524"}
Sep 18 21:52:56.103636 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:56.103606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6jk4g" event={"ID":"933fc10b-a510-420e-8cae-3a99f8061cd5","Type":"ContainerStarted","Data":"bb59035da493b7e8c36c04bfef4efdbfcd8f3cb685d629930e987c6f1204ed68"}
Sep 18 21:52:56.105337 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:56.105308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm" event={"ID":"b4877d2e-358b-40b0-be2a-c86b2196e7ac","Type":"ContainerStarted","Data":"2475a5b3310dfa5b92803c671acb75865f98dd57190c64be119e9d614894e364"}
Sep 18 21:52:56.105464 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:56.105355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm" event={"ID":"b4877d2e-358b-40b0-be2a-c86b2196e7ac","Type":"ContainerStarted","Data":"aca53948ab147cd070956fd735da8956f34edd4820732e3d1e309349f955153c"}
Sep 18 21:52:56.106560 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:56.106539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c" event={"ID":"865926b3-b843-49ce-bd12-591aa6b8c33a","Type":"ContainerStarted","Data":"32341394e645df5775a7c4cec1d08d1f2e16b10e6d2d4e4bed82d439cae0aab5"}
Sep 18 21:52:56.122823 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:56.122771    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-2ddwm" podStartSLOduration=1.122753045 podStartE2EDuration="1.122753045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:56.122139678 +0000 UTC m=+154.978795331" watchObservedRunningTime="2026-09-18 21:52:56.122753045 +0000 UTC m=+154.979408698"
Sep 18 21:52:56.749148 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:56.749112    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:56.931926 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:56.931891    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:57.014319 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:57.014206    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:57.116189 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:57.116157    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6jk4g" event={"ID":"933fc10b-a510-420e-8cae-3a99f8061cd5","Type":"ContainerStarted","Data":"ecbf3fd2a2f6c0247f3aa6591f9b6cc32aeb0740138a9f1505aaa4a05107f685"}
Sep 18 21:52:57.117807 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:57.117782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q" event={"ID":"f3133e4f-8bf5-41fe-8467-ca57a8a1a309","Type":"ContainerStarted","Data":"c53c820a996ed07b19fdf07288d800a028f4ae98b85eacb5ee947f277c5f4608"}
Sep 18 21:52:57.140111 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:57.140066    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6jk4g" podStartSLOduration=33.889151463 podStartE2EDuration="36.140053145s" totalImagesPullingTime="2.250901682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:52:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:57.138788604 +0000 UTC m=+155.995444272" watchObservedRunningTime="2026-09-18 21:52:57.140053145 +0000 UTC m=+155.996708797"
Sep 18 21:52:57.156528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:57.156483    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-v598q" podStartSLOduration=0.963039789 podStartE2EDuration="2.156464453s" totalImagesPullingTime="1.193424664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:57.155855834 +0000 UTC m=+156.012511467" watchObservedRunningTime="2026-09-18 21:52:57.156464453 +0000 UTC m=+156.013120107"
Sep 18 21:52:57.927700 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:57.927666    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:58.003899 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:58.003865    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh" podUID="ec02faa2-62b4-4c66-9ffd-6b24ae4d9259"
Sep 18 21:52:58.046261 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:58.046234    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tgn5c" podUID="070d51d0-7645-40a7-a7e8-9d1a9376b87d"
Sep 18 21:52:58.122524 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:58.122497    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c" event={"ID":"865926b3-b843-49ce-bd12-591aa6b8c33a","Type":"ContainerStarted","Data":"53ed61ae0554f4e539959bb1a3de2e3ea271597d77cdb6a7822216a1a56f85d8"}
Sep 18 21:52:58.122652 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:58.122627    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:52:58.122841 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:58.122823    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"
Sep 18 21:52:58.122999 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:58.122982    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:52:58.127449 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:58.127432    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c"
Sep 18 21:52:58.140452 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:58.140417    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c" podStartSLOduration=1.639555131 podStartE2EDuration="3.140404712s" totalImagesPullingTime="1.500849581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:58.139779575 +0000 UTC m=+156.996435228" watchObservedRunningTime="2026-09-18 21:52:58.140404712 +0000 UTC m=+156.997060364"
Sep 18 21:52:58.303091 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:58.303060    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:58.401249 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:58.401224    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:58.981800 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:58.981765    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:59.041382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.041352    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"]
Sep 18 21:52:59.044482 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.044466    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.047892 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.047872    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 21:52:59.048003 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.047884    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l4972\""
Sep 18 21:52:59.048003 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.047965    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 21:52:59.048168 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.048155    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 21:52:59.060219 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.060195    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"]
Sep 18 21:52:59.086563 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.086539    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:59.115648 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.115626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.115746 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.115687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55v4\" (UniqueName: \"kubernetes.io/projected/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-kube-api-access-n55v4\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.115781 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.115745    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.115781 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.115767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.164607 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.164567    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:52:59.216403 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.216375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.216546 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.216463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55v4\" (UniqueName: \"kubernetes.io/projected/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-kube-api-access-n55v4\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.216546 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.216508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.216546 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.216535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.217125 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.217100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.218699 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.218671    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.218775 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.218763    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.225125 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.225097    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55v4\" (UniqueName: \"kubernetes.io/projected/ce747ee0-3d50-49bb-9f7c-08f70793d5e1-kube-api-access-n55v4\") pod \"prometheus-operator-7bd5d56d6-pk5s2\" (UID: \"ce747ee0-3d50-49bb-9f7c-08f70793d5e1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.353522 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.353482    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"
Sep 18 21:52:59.626082 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:52:59.626021    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2"]
Sep 18 21:52:59.629731 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:52:59.629707    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce747ee0_3d50_49bb_9f7c_08f70793d5e1.slice/crio-64447caf37ff53f08c4a7130c6d7e722595d06b4e18089042015d85c8ec1a0a7 WatchSource:0}: Error finding container 64447caf37ff53f08c4a7130c6d7e722595d06b4e18089042015d85c8ec1a0a7: Status 404 returned error can't find the container with id 64447caf37ff53f08c4a7130c6d7e722595d06b4e18089042015d85c8ec1a0a7
Sep 18 21:52:59.631838 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:52:59.631791    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vqfxz" podUID="c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba"
Sep 18 21:53:00.129250 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:00.129219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2" event={"ID":"ce747ee0-3d50-49bb-9f7c-08f70793d5e1","Type":"ContainerStarted","Data":"64447caf37ff53f08c4a7130c6d7e722595d06b4e18089042015d85c8ec1a0a7"}
Sep 18 21:53:01.134219 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:01.134171    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2" event={"ID":"ce747ee0-3d50-49bb-9f7c-08f70793d5e1","Type":"ContainerStarted","Data":"3f9d302c129003bbd3352e0e96d789c5176c1361561beeba4d63e72a99cadca4"}
Sep 18 21:53:01.134612 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:01.134287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2" event={"ID":"ce747ee0-3d50-49bb-9f7c-08f70793d5e1","Type":"ContainerStarted","Data":"3e286ce6e1d2ca59d0e019178c2933ba81795e7a728dd91d15c3fea9c3615386"}
Sep 18 21:53:01.140211 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:01.140177    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:01.152839 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:01.152800    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-pk5s2" podStartSLOduration=0.947780682 podStartE2EDuration="2.15278437s" totalImagesPullingTime="1.205003688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:01.151584994 +0000 UTC m=+160.008240647" watchObservedRunningTime="2026-09-18 21:53:01.15278437 +0000 UTC m=+160.009440021"
Sep 18 21:53:02.942618 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:02.942579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:53:02.943063 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:02.942660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:53:02.944921 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:02.944899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/070d51d0-7645-40a7-a7e8-9d1a9376b87d-metrics-tls\") pod \"dns-default-tgn5c\" (UID: \"070d51d0-7645-40a7-a7e8-9d1a9376b87d\") " pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:53:02.945024 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:02.944935    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"image-registry-7d8786bcc9-7w7lh\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") " pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:53:03.195334 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.195236    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:03.225618 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.225592    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gb4w8\""
Sep 18 21:53:03.226238 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.226221    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xmfvd\""
Sep 18 21:53:03.234259 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.234240    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:53:03.234361 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.234321    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:53:03.442040 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.442005    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:03.488759 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.488693    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nv98w"]
Sep 18 21:53:03.493400 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.493380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.497757 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.497735    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 21:53:03.498046 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.498012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mpctd\""
Sep 18 21:53:03.498564 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.498549    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 21:53:03.498637 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.498584    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 21:53:03.524845 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.524821    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tgn5c"]
Sep 18 21:53:03.527027 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:53:03.527003    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070d51d0_7645_40a7_a7e8_9d1a9376b87d.slice/crio-5eee29aa2fcf3d24631cfd4e9dff4498239bc1f0e29961eb66243ad09043e615 WatchSource:0}: Error finding container 5eee29aa2fcf3d24631cfd4e9dff4498239bc1f0e29961eb66243ad09043e615: Status 404 returned error can't find the container with id 5eee29aa2fcf3d24631cfd4e9dff4498239bc1f0e29961eb66243ad09043e615
Sep 18 21:53:03.539831 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.539810    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d8786bcc9-7w7lh"]
Sep 18 21:53:03.542028 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:53:03.542007    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec02faa2_62b4_4c66_9ffd_6b24ae4d9259.slice/crio-229c0ba096eba990a186621b9204e048806204e05fdc3811513178ef0ba284fa WatchSource:0}: Error finding container 229c0ba096eba990a186621b9204e048806204e05fdc3811513178ef0ba284fa: Status 404 returned error can't find the container with id 229c0ba096eba990a186621b9204e048806204e05fdc3811513178ef0ba284fa
Sep 18 21:53:03.547935 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.547916    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-sys\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.548027 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.547947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-textfile\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.548027 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.547969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-root\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.548124 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.548070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnr26\" (UniqueName: \"kubernetes.io/projected/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-kube-api-access-lnr26\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.548168 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.548114    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.548168 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.548146    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.548168 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.548164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-wtmp\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.548283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.548185    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-metrics-client-ca\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.548283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.548237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-tls\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.565228 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.565192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:03.649043 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-sys\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649143 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649052    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-textfile\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649143 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-root\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649143 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-root\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649305 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnr26\" (UniqueName: \"kubernetes.io/projected/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-kube-api-access-lnr26\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649305 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649305 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649305 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-wtmp\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649305 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649207    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-sys\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649305 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-metrics-client-ca\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649305 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649286    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-tls\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649657 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649322    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-wtmp\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649657 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649440    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-textfile\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.649943 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-metrics-client-ca\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.650028 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.649956    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.651404 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.651379    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.651503 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.651487    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-node-exporter-tls\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.661315 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.661298    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnr26\" (UniqueName: \"kubernetes.io/projected/f5f7c83d-684c-4118-8d5d-443f8cdb46ee-kube-api-access-lnr26\") pod \"node-exporter-nv98w\" (UID: \"f5f7c83d-684c-4118-8d5d-443f8cdb46ee\") " pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.689548 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.689520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:03.800493 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.800447    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:03.803208 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:03.803188    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nv98w"
Sep 18 21:53:03.819635 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:53:03.819609    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f7c83d_684c_4118_8d5d_443f8cdb46ee.slice/crio-c4eff91e919eb3ea545b8f241b3df2d18c9e04c2c9d95f4bb094fa6cf143d49f WatchSource:0}: Error finding container c4eff91e919eb3ea545b8f241b3df2d18c9e04c2c9d95f4bb094fa6cf143d49f: Status 404 returned error can't find the container with id c4eff91e919eb3ea545b8f241b3df2d18c9e04c2c9d95f4bb094fa6cf143d49f
Sep 18 21:53:04.066508 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.066414    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:04.146188 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.146159    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tgn5c" event={"ID":"070d51d0-7645-40a7-a7e8-9d1a9376b87d","Type":"ContainerStarted","Data":"5eee29aa2fcf3d24631cfd4e9dff4498239bc1f0e29961eb66243ad09043e615"}
Sep 18 21:53:04.147814 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.147786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv98w" event={"ID":"f5f7c83d-684c-4118-8d5d-443f8cdb46ee","Type":"ContainerStarted","Data":"c4eff91e919eb3ea545b8f241b3df2d18c9e04c2c9d95f4bb094fa6cf143d49f"}
Sep 18 21:53:04.149611 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.149590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh" event={"ID":"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259","Type":"ContainerStarted","Data":"8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d"}
Sep 18 21:53:04.149726 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.149628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh" event={"ID":"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259","Type":"ContainerStarted","Data":"229c0ba096eba990a186621b9204e048806204e05fdc3811513178ef0ba284fa"}
Sep 18 21:53:04.149726 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.149679    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:53:04.176808 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.176762    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh" podStartSLOduration=161.176749136 podStartE2EDuration="2m41.176749136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:04.17496385 +0000 UTC m=+163.031619502" watchObservedRunningTime="2026-09-18 21:53:04.176749136 +0000 UTC m=+163.033404788"
Sep 18 21:53:04.484857 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.484768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:04.622521 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.622364    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:04.653842 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.653035    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:53:04.657358 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.657329    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.661679 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.661647    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 21:53:04.661960 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.661938    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 21:53:04.664838 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.664816    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 21:53:04.665389 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.665366    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 21:53:04.665482 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.665430    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 21:53:04.665790 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.665772    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 21:53:04.666118 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.666100    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 21:53:04.666118 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.666113    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 21:53:04.666886 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.666306    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 21:53:04.666886 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.666602    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pbqfr\""
Sep 18 21:53:04.683115 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.683006    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:53:04.731798 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.731766    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:04.758175 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758327 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758185    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758327 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758207    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhwc\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-kube-api-access-jbhwc\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758327 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758327 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758327 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758552 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758350    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758552 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-out\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758552 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758552 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758774 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758575    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758774 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-web-config\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.758774 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.758676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.859622 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-web-config\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.859804 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.859804 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.859804 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.859804 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhwc\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-kube-api-access-jbhwc\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.859804 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.860074 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.860074 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.860074 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.860074 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-out\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.860074 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.860074 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.859996    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.860074 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.860040    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.862398 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.860766    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.862398 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.862039    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.862398 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.862113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.863712 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.863670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.864985 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.864956    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.866212 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.866191    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-out\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.866346 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.866325    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-web-config\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.866567 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.866549    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.866747 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.866727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.867076 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.867054    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.867423 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.867401    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.868520 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.868493    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.870070 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.870030    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhwc\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-kube-api-access-jbhwc\") pod \"alertmanager-main-0\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:04.968457 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:04.968424    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:53:05.604671 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:05.604647    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:53:05.605613 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:53:05.605589    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b78e70f_e99f_4c91_9c94_1bd0d82a7ccb.slice/crio-9ae0d1d01f648953388c9b42355741e1e41f3e41e627da588e43f3c8dda95f7f WatchSource:0}: Error finding container 9ae0d1d01f648953388c9b42355741e1e41f3e41e627da588e43f3c8dda95f7f: Status 404 returned error can't find the container with id 9ae0d1d01f648953388c9b42355741e1e41f3e41e627da588e43f3c8dda95f7f
Sep 18 21:53:06.159337 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.159294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tgn5c" event={"ID":"070d51d0-7645-40a7-a7e8-9d1a9376b87d","Type":"ContainerStarted","Data":"f7256b6bf74c2b571b6c14d48e444f602f2eff4eb1df5f0825331f98bcbfd739"}
Sep 18 21:53:06.159516 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.159350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tgn5c" event={"ID":"070d51d0-7645-40a7-a7e8-9d1a9376b87d","Type":"ContainerStarted","Data":"b4a7399fbbb43d93ae8e1dbe0df8028d0b7c4854ff3e034d6d0e62f8f070a474"}
Sep 18 21:53:06.159516 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.159428    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:53:06.161116 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.161090    2564 generic.go:356] "Generic (PLEG): container finished" podID="f5f7c83d-684c-4118-8d5d-443f8cdb46ee" containerID="8e340fc30fb4b93e71c475ea3fb5773dd0c2429280ebdc05501d381c29b10c24" exitCode=0
Sep 18 21:53:06.161218 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.161169    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv98w" event={"ID":"f5f7c83d-684c-4118-8d5d-443f8cdb46ee","Type":"ContainerDied","Data":"8e340fc30fb4b93e71c475ea3fb5773dd0c2429280ebdc05501d381c29b10c24"}
Sep 18 21:53:06.162442 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.162424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerStarted","Data":"9ae0d1d01f648953388c9b42355741e1e41f3e41e627da588e43f3c8dda95f7f"}
Sep 18 21:53:06.182065 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.182025    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tgn5c" podStartSLOduration=130.39484978 podStartE2EDuration="2m12.18201257s" totalImagesPullingTime="1.78716279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:06.180871068 +0000 UTC m=+165.037526719" watchObservedRunningTime="2026-09-18 21:53:06.18201257 +0000 UTC m=+165.038668221"
Sep 18 21:53:06.324000 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.323950    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:06.427609 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.427421    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:06.516033 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.515998    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:06.846759 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.846733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:06.948703 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:06.948675    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:07.168203 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.168172    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv98w" event={"ID":"f5f7c83d-684c-4118-8d5d-443f8cdb46ee","Type":"ContainerStarted","Data":"58debe09f22e8768948e33dcb05894c50e662698d5b27a1a275c81b7b4052a3f"}
Sep 18 21:53:07.168383 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.168285    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv98w" event={"ID":"f5f7c83d-684c-4118-8d5d-443f8cdb46ee","Type":"ContainerStarted","Data":"51357be3d6215966f93085111922a2f0835b20b27172cb5cdee597a82d1f7b9b"}
Sep 18 21:53:07.169725 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.169705    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerID="d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0" exitCode=0
Sep 18 21:53:07.169840 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.169762    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerDied","Data":"d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0"}
Sep 18 21:53:07.190094 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.190056    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nv98w" podStartSLOduration=2.629723834 podStartE2EDuration="4.190043652s" totalImagesPullingTime="1.488694799s" totalInitContainerRuntime="71.625019ms" isStatefulPod=true podCreationTimestamp="2026-09-18 21:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:07.189414923 +0000 UTC m=+166.046070565" watchObservedRunningTime="2026-09-18 21:53:07.190043652 +0000 UTC m=+166.046699350"
Sep 18 21:53:07.870675 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.870641    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"]
Sep 18 21:53:07.873972 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.873952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:07.880470 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.880444    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 21:53:07.880645 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.880608    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 21:53:07.881398 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.881206    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tsldk\""
Sep 18 21:53:07.881398 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.881360    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 21:53:07.884225 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.884196    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 21:53:07.884529 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.884513    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-16asmm7ikslcu\""
Sep 18 21:53:07.893535 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.893514    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"]
Sep 18 21:53:07.985310 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.985258    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l4xm\" (UniqueName: \"kubernetes.io/projected/4c3fc516-ff87-4c33-a504-9433fe34f0a7-kube-api-access-6l4xm\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:07.985429 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.985324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4c3fc516-ff87-4c33-a504-9433fe34f0a7-secret-metrics-server-client-certs\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:07.985429 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.985370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4c3fc516-ff87-4c33-a504-9433fe34f0a7-secret-metrics-server-tls\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:07.985514 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.985438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3fc516-ff87-4c33-a504-9433fe34f0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:07.985514 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.985475    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4c3fc516-ff87-4c33-a504-9433fe34f0a7-metrics-server-audit-profiles\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:07.985514 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.985504    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4c3fc516-ff87-4c33-a504-9433fe34f0a7-client-ca-bundle\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:07.985650 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:07.985548    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4c3fc516-ff87-4c33-a504-9433fe34f0a7-audit-log\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.086602 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.086570    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l4xm\" (UniqueName: \"kubernetes.io/projected/4c3fc516-ff87-4c33-a504-9433fe34f0a7-kube-api-access-6l4xm\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.086602 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.086606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4c3fc516-ff87-4c33-a504-9433fe34f0a7-secret-metrics-server-client-certs\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.086830 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.086632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4c3fc516-ff87-4c33-a504-9433fe34f0a7-secret-metrics-server-tls\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.086830 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.086666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3fc516-ff87-4c33-a504-9433fe34f0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.086830 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.086696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4c3fc516-ff87-4c33-a504-9433fe34f0a7-metrics-server-audit-profiles\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.086830 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.086730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4c3fc516-ff87-4c33-a504-9433fe34f0a7-client-ca-bundle\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.086830 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.086772    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4c3fc516-ff87-4c33-a504-9433fe34f0a7-audit-log\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.087241 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.087195    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4c3fc516-ff87-4c33-a504-9433fe34f0a7-audit-log\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.087738 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.087713    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3fc516-ff87-4c33-a504-9433fe34f0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.087878 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.087860    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4c3fc516-ff87-4c33-a504-9433fe34f0a7-metrics-server-audit-profiles\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.089382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.089343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4c3fc516-ff87-4c33-a504-9433fe34f0a7-secret-metrics-server-tls\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.089530 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.089507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4c3fc516-ff87-4c33-a504-9433fe34f0a7-secret-metrics-server-client-certs\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.089706 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.089684    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4c3fc516-ff87-4c33-a504-9433fe34f0a7-client-ca-bundle\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.102476 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.102455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l4xm\" (UniqueName: \"kubernetes.io/projected/4c3fc516-ff87-4c33-a504-9433fe34f0a7-kube-api-access-6l4xm\") pod \"metrics-server-7cc9c9ffcf-xttz5\" (UID: \"4c3fc516-ff87-4c33-a504-9433fe34f0a7\") " pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.185956 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.185873    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:08.220380 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.220283    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:08.803200 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.803025    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"]
Sep 18 21:53:08.805614 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:53:08.805587    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3fc516_ff87_4c33_a504_9433fe34f0a7.slice/crio-3657a5df155fab65cde8ec9054ce866d01cbdb3031a6ad56573a91d3bf6cb9a1 WatchSource:0}: Error finding container 3657a5df155fab65cde8ec9054ce866d01cbdb3031a6ad56573a91d3bf6cb9a1: Status 404 returned error can't find the container with id 3657a5df155fab65cde8ec9054ce866d01cbdb3031a6ad56573a91d3bf6cb9a1
Sep 18 21:53:08.982946 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:08.982907    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:09.180737 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.180696    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerStarted","Data":"656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868"}
Sep 18 21:53:09.180914 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.180779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerStarted","Data":"ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28"}
Sep 18 21:53:09.180914 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.180815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerStarted","Data":"7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa"}
Sep 18 21:53:09.180914 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.180845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerStarted","Data":"212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78"}
Sep 18 21:53:09.180914 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.180878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerStarted","Data":"751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b"}
Sep 18 21:53:09.181884 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.181857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5" event={"ID":"4c3fc516-ff87-4c33-a504-9433fe34f0a7","Type":"ContainerStarted","Data":"3657a5df155fab65cde8ec9054ce866d01cbdb3031a6ad56573a91d3bf6cb9a1"}
Sep 18 21:53:09.793320 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.793292    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:53:09.798533 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.798048    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.799680 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.799619    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:09.804430 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.804410    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-59epskovr0sfc\""
Sep 18 21:53:09.804908 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.804887    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 21:53:09.805180 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.805051    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 21:53:09.805657 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.805561    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 21:53:09.805760 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.805747    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 21:53:09.806066 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.806047    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 21:53:09.806564 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.806446    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 21:53:09.806757 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.806685    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 21:53:09.807057 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.806854    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 21:53:09.807057 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.807016    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 21:53:09.807202 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.807171    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 21:53:09.808058 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.807373    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 21:53:09.808058 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.807570    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mnm42\""
Sep 18 21:53:09.808058 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.807898    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 21:53:09.808249 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.808103    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 21:53:09.811067 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.811034    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 21:53:09.820628 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.820605    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:53:09.904442 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn2g6\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-kube-api-access-rn2g6\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904597 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904453    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-config\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904597 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904597 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-config-out\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904722 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904722 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904722 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904722 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904869 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904869 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904781    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904869 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904966 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.904966 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.905049 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.904964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.905049 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.905003    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.905049 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.905035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.905153 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.905055    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.905153 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.905083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.905153 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.905115    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:09.905153 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:09.905132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-web-config\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.006569 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.006569 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006364    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.006569 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006608    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-web-config\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn2g6\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-kube-api-access-rn2g6\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-config\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-config-out\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.006974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.007016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.007047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.007085    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.007120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.007117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.008086 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.007616    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.008086 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.007975    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.008768 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.008737    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.009222 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.009198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.009727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.009624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.010653 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.010124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.010653 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.010321    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-web-config\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.010653 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.010323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-config-out\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.011008 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.010955    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.012046 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.011630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-config\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.012046 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.011917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.012485 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.012378    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.012627 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.012584    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.012734 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.012677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.013167 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.013125    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.014989 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.014938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.015194 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.015146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.015390 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.015371    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.017017 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.016999    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.018565 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.018536    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn2g6\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-kube-api-access-rn2g6\") pod \"prometheus-k8s-0\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.111208 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.111175    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:10.194429 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.194363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerStarted","Data":"a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a"}
Sep 18 21:53:10.231237 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.231172    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.860529058 podStartE2EDuration="6.231148471s" totalImagesPullingTime="3.354936245s" totalInitContainerRuntime="15.683168ms" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:10.229202046 +0000 UTC m=+169.085857717" watchObservedRunningTime="2026-09-18 21:53:10.231148471 +0000 UTC m=+169.087804125"
Sep 18 21:53:10.565930 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:10.565904    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:53:10.567417 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:53:10.567391    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84044821_3536_4a4d_8c78_31403988ae43.slice/crio-f2211178f93299196b2efdf3078213e39cdd0f14d9be1501c8cb33c54c22a4fa WatchSource:0}: Error finding container f2211178f93299196b2efdf3078213e39cdd0f14d9be1501c8cb33c54c22a4fa: Status 404 returned error can't find the container with id f2211178f93299196b2efdf3078213e39cdd0f14d9be1501c8cb33c54c22a4fa
Sep 18 21:53:11.200716 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:11.200681    2564 generic.go:356] "Generic (PLEG): container finished" podID="84044821-3536-4a4d-8c78-31403988ae43" containerID="aba12d36043cc6a83165881003ff430f7681662d2e3856affbc6514f5adcafc2" exitCode=0
Sep 18 21:53:11.201104 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:11.200768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerDied","Data":"aba12d36043cc6a83165881003ff430f7681662d2e3856affbc6514f5adcafc2"}
Sep 18 21:53:11.201104 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:11.200882    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerStarted","Data":"f2211178f93299196b2efdf3078213e39cdd0f14d9be1501c8cb33c54c22a4fa"}
Sep 18 21:53:11.202418 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:11.202386    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5" event={"ID":"4c3fc516-ff87-4c33-a504-9433fe34f0a7","Type":"ContainerStarted","Data":"b2e45b1d6d34bd5271a919a64ee4538c0616d7107321d179db55de50e84f1ff3"}
Sep 18 21:53:11.276857 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:11.276819    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5" podStartSLOduration=2.749022255 podStartE2EDuration="4.276805902s" totalImagesPullingTime="1.527783647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:11.275873591 +0000 UTC m=+170.132529244" watchObservedRunningTime="2026-09-18 21:53:11.276805902 +0000 UTC m=+170.133461554"
Sep 18 21:53:11.924945 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:11.924913    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:12.025056 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:12.025017    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:12.179312 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:12.179213    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:12.346655 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:12.346628    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:12.412690 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:12.412650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:12.752014 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:12.751973    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:12.836251 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:12.836208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:13.187156 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:13.187116    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:13.382984 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:13.382931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:13.576017 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:13.575981    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:13.613854 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:13.613822    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:53:13.681251 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:13.681216    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:13.784482 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:13.784443    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:13.879882 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:13.879784    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:14.218673 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:14.218648    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerStarted","Data":"c4b4b6417a734d214d7bd74e1b4d02b87e653f140f9f44cc9fa4ac64c941ab46"}
Sep 18 21:53:14.643611 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:14.643575    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:14.803112 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:14.803078    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:15.225975 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:15.225925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerStarted","Data":"0de09efb01fa59b22fbbfdbb1ce0add56c937ebb8ed400e407843c14bd8ebde7"}
Sep 18 21:53:16.171689 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:16.171668    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tgn5c"
Sep 18 21:53:16.234110 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:16.234035    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerStarted","Data":"4daeef37483dc4ed4b6ec28f1bdd0f823abee88a558bf80bc9ce269caab4ef52"}
Sep 18 21:53:16.234110 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:16.234093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerStarted","Data":"89dcc0ac750d0c4f99ba985b9a2f71f121998ec350ebb668d120b5789f377b90"}
Sep 18 21:53:16.234319 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:16.234120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerStarted","Data":"b14743487f28537a824262a3c1151ed9bf44b6ca2755f3dd0960efa8d32b6fc6"}
Sep 18 21:53:16.234319 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:16.234143    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerStarted","Data":"e8e6cd321068a7f580f8a176a3a31ad5e0b3744ec7602429a97a0fef4bad4618"}
Sep 18 21:53:16.276332 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:16.276256    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.64048052 podStartE2EDuration="7.276236235s" totalImagesPullingTime="4.612897139s" totalInitContainerRuntime="22.858576ms" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:16.274304488 +0000 UTC m=+175.130960139" watchObservedRunningTime="2026-09-18 21:53:16.276236235 +0000 UTC m=+175.132891897"
Sep 18 21:53:17.363054 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:17.363023    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d8786bcc9-7w7lh"]
Sep 18 21:53:17.464680 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:17.464638    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:17.567764 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:17.567731    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:17.669740 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:17.669649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:20.111488 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:20.111454    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:53:21.320279 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:21.320222    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:21.419937 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:21.419899    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:24.394845 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:24.394808    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:24.465382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:24.465344    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:25.385338 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:25.385306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:26.027470 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:26.027430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:27.368467 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:27.368440    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:53:28.186803 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:28.186772    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:28.186967 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:28.186824    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:29.716770 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:29.716733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:29.818420 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:29.818395    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:29.891424 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:29.891399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:32.178228 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:32.178198    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:32.273662 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:32.273631    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:32.350132 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:32.350100    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:32.450676 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:32.450605    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:32.524052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:32.524022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:37.811231 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:37.811193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:37.906255 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:37.906223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:38.888418 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:38.888386    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:39.066389 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:39.066349    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:39.233751 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:39.233665    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:39.386193 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:39.386161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:39.617124 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:39.617086    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:39.722437 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:39.722406    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:42.382177 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.382141    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh" podUID="ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" containerName="registry" containerID="cri-o://8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d" gracePeriod=30
Sep 18 21:53:42.627881 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.627857    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:53:42.685386 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.685321    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-installation-pull-secrets\") pod \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") "
Sep 18 21:53:42.685386 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.685360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-image-registry-private-configuration\") pod \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") "
Sep 18 21:53:42.685386 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.685386    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-trusted-ca\") pod \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") "
Sep 18 21:53:42.685584 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.685417    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tklws\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-kube-api-access-tklws\") pod \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") "
Sep 18 21:53:42.685584 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.685459    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") pod \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") "
Sep 18 21:53:42.685584 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.685528    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-ca-trust-extracted\") pod \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") "
Sep 18 21:53:42.685584 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.685550    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-bound-sa-token\") pod \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") "
Sep 18 21:53:42.685788 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.685592    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-certificates\") pod \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\" (UID: \"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259\") "
Sep 18 21:53:42.685912 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.685770    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:53:42.686168 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.686145    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:53:42.687938 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.687905    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:53:42.688036 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.688016    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:53:42.688110 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.688051    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-kube-api-access-tklws" (OuterVolumeSpecName: "kube-api-access-tklws") pod "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259"). InnerVolumeSpecName "kube-api-access-tklws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:53:42.688110 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.688058    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:53:42.688177 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.688127    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:53:42.693740 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.693716    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" (UID: "ec02faa2-62b4-4c66-9ffd-6b24ae4d9259"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:53:42.786184 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.786154    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tklws\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-kube-api-access-tklws\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:53:42.786184 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.786184    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:53:42.786184 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.786194    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-ca-trust-extracted\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:53:42.786385 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.786203    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-bound-sa-token\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:53:42.786385 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.786212    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-registry-certificates\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:53:42.786385 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.786220    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-installation-pull-secrets\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:53:42.786385 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.786229    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-image-registry-private-configuration\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:53:42.786385 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:42.786238    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259-trusted-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:53:43.341642 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.341613    2564 generic.go:356] "Generic (PLEG): container finished" podID="ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" containerID="8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d" exitCode=0
Sep 18 21:53:43.341812 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.341653    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh"
Sep 18 21:53:43.341812 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.341690    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh" event={"ID":"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259","Type":"ContainerDied","Data":"8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d"}
Sep 18 21:53:43.341812 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.341772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d8786bcc9-7w7lh" event={"ID":"ec02faa2-62b4-4c66-9ffd-6b24ae4d9259","Type":"ContainerDied","Data":"229c0ba096eba990a186621b9204e048806204e05fdc3811513178ef0ba284fa"}
Sep 18 21:53:43.341812 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.341804    2564 scope.go:117] "RemoveContainer" containerID="8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d"
Sep 18 21:53:43.350441 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.350425    2564 scope.go:117] "RemoveContainer" containerID="8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d"
Sep 18 21:53:43.350720 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:53:43.350701    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d\": container with ID starting with 8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d not found: ID does not exist" containerID="8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d"
Sep 18 21:53:43.350792 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.350733    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d"} err="failed to get container status \"8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d\": rpc error: code = NotFound desc = could not find container \"8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d\": container with ID starting with 8920f9da6cc3241676497beb854937dfdfda4aabb4bae91bc07ec2736fcc986d not found: ID does not exist"
Sep 18 21:53:43.368831 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.368808    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d8786bcc9-7w7lh"]
Sep 18 21:53:43.377069 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.377051    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d8786bcc9-7w7lh"]
Sep 18 21:53:43.617894 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:43.617819    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" path="/var/lib/kubelet/pods/ec02faa2-62b4-4c66-9ffd-6b24ae4d9259/volumes"
Sep 18 21:53:48.192121 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:48.192091    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:48.195773 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:48.195756    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cc9c9ffcf-xttz5"
Sep 18 21:53:51.331411 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:51.331377    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:51.435402 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:51.435367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:51.538008 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:51.537982    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:53.959783 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:53.959745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:54.050528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:54.050494    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:56.394703 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:56.394675    2564 generic.go:356] "Generic (PLEG): container finished" podID="211fed90-33df-4a21-af33-482fd0873638" containerID="728f9a7a279e5e5a028b5ed4afc0a8b1c1c33fc9afb98e337a9faf337a29dfd0" exitCode=0
Sep 18 21:53:56.395104 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:56.394753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf" event={"ID":"211fed90-33df-4a21-af33-482fd0873638","Type":"ContainerDied","Data":"728f9a7a279e5e5a028b5ed4afc0a8b1c1c33fc9afb98e337a9faf337a29dfd0"}
Sep 18 21:53:56.395144 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:56.395117    2564 scope.go:117] "RemoveContainer" containerID="728f9a7a279e5e5a028b5ed4afc0a8b1c1c33fc9afb98e337a9faf337a29dfd0"
Sep 18 21:53:56.770984 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:56.770906    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:56.839577 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:56.839545    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:53:57.400226 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:53:57.400193    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ws6nf" event={"ID":"211fed90-33df-4a21-af33-482fd0873638","Type":"ContainerStarted","Data":"b7d964849cca099e1a4b4a8b36e8257650cce82d4e0a20adab069145cba59c45"}
Sep 18 21:54:06.329735 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:06.329701    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:06.418116 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:06.418074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:06.682875 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:06.682792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:06.783700 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:06.783673    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:10.113748 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:10.112320    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:10.134191 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:10.134166    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:10.472985 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:10.472909    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:20.959571 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:20.959532    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:22.859526 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:22.859492    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:23.541820 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:23.541789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:23.861670 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:23.861584    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:54:23.862109 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:23.861857    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="alertmanager" containerID="cri-o://751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b" gracePeriod=120
Sep 18 21:54:23.862109 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:23.861866    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="prom-label-proxy" containerID="cri-o://a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a" gracePeriod=120
Sep 18 21:54:23.862109 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:23.861871    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy" containerID="cri-o://ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28" gracePeriod=120
Sep 18 21:54:23.862109 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:23.861910    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy-web" containerID="cri-o://7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa" gracePeriod=120
Sep 18 21:54:23.862109 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:23.861946    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="config-reloader" containerID="cri-o://212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78" gracePeriod=120
Sep 18 21:54:23.862109 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:23.862011    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy-metric" containerID="cri-o://656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868" gracePeriod=120
Sep 18 21:54:24.513882 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.513849    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerID="a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a" exitCode=0
Sep 18 21:54:24.513882 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.513873    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerID="656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868" exitCode=0
Sep 18 21:54:24.513882 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.513881    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerID="ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28" exitCode=0
Sep 18 21:54:24.513882 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.513886    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerID="212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78" exitCode=0
Sep 18 21:54:24.513882 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.513891    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerID="751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b" exitCode=0
Sep 18 21:54:24.514170 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.513921    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerDied","Data":"a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a"}
Sep 18 21:54:24.514170 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.513979    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerDied","Data":"656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868"}
Sep 18 21:54:24.514170 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.514005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerDied","Data":"ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28"}
Sep 18 21:54:24.514170 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.514027    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerDied","Data":"212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78"}
Sep 18 21:54:24.514170 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:24.514049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerDied","Data":"751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b"}
Sep 18 21:54:25.110357 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.110329    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.229892 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.229800    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-main-tls\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.229892 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.229849    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.229892 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.229875    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-tls-assets\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.229892 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.229893    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbhwc\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-kube-api-access-jbhwc\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.230214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.229910    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.230214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.229937    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-cluster-tls-config\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.230214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.229955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.230214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.229979    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-main-db\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.230214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.230004    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-out\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.230214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.230049    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-trusted-ca-bundle\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.230214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.230075    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-volume\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.230214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.230119    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-web-config\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.230214 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.230143    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-metrics-client-ca\") pod \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\" (UID: \"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb\") "
Sep 18 21:54:25.231135 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.230802    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:25.231135 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.231025    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:25.231645 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.231399    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:54:25.232903 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.232867    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:54:25.233006 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.232909    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-kube-api-access-jbhwc" (OuterVolumeSpecName: "kube-api-access-jbhwc") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "kube-api-access-jbhwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:54:25.233714 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.233679    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:25.233813 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.233790    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:25.234030 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.233992    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-volume" (OuterVolumeSpecName: "config-volume") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:25.234108 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.234075    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:25.234153 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.234099    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:25.234544 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.234522    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-out" (OuterVolumeSpecName: "config-out") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:54:25.238433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.238256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:25.243807 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.243785    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-web-config" (OuterVolumeSpecName: "web-config") pod "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" (UID: "7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:25.331563 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331515    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-main-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331563 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331557    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331563 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331567    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-tls-assets\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331563 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331576    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbhwc\" (UniqueName: \"kubernetes.io/projected/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-kube-api-access-jbhwc\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331563 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331585    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331867 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331594    2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-cluster-tls-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331867 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331602    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331867 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331612    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-main-db\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331867 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331620    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-out\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331867 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331628    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331867 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331636    2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-config-volume\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331867 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331644    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-web-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.331867 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.331651    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb-metrics-client-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:25.522872 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.522783    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerID="7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa" exitCode=0
Sep 18 21:54:25.522872 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.522846    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerDied","Data":"7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa"}
Sep 18 21:54:25.522872 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.522861    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.523185 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.522924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb","Type":"ContainerDied","Data":"9ae0d1d01f648953388c9b42355741e1e41f3e41e627da588e43f3c8dda95f7f"}
Sep 18 21:54:25.523185 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.522971    2564 scope.go:117] "RemoveContainer" containerID="a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a"
Sep 18 21:54:25.531387 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.531366    2564 scope.go:117] "RemoveContainer" containerID="656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868"
Sep 18 21:54:25.542461 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.542439    2564 scope.go:117] "RemoveContainer" containerID="ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28"
Sep 18 21:54:25.549696 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.549665    2564 scope.go:117] "RemoveContainer" containerID="7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa"
Sep 18 21:54:25.553675 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.553647    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:54:25.556996 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.556970    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:54:25.559255 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.559230    2564 scope.go:117] "RemoveContainer" containerID="212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78"
Sep 18 21:54:25.567097 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.567074    2564 scope.go:117] "RemoveContainer" containerID="751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b"
Sep 18 21:54:25.574820 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.574794    2564 scope.go:117] "RemoveContainer" containerID="d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0"
Sep 18 21:54:25.583092 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.583068    2564 scope.go:117] "RemoveContainer" containerID="a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a"
Sep 18 21:54:25.583441 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:54:25.583420    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a\": container with ID starting with a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a not found: ID does not exist" containerID="a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a"
Sep 18 21:54:25.583494 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.583452    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a"} err="failed to get container status \"a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a\": rpc error: code = NotFound desc = could not find container \"a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a\": container with ID starting with a9fc422b91813b5f8004898c956fac4c147ecf51e103a96c09f78d7b37e92a4a not found: ID does not exist"
Sep 18 21:54:25.583494 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.583472    2564 scope.go:117] "RemoveContainer" containerID="656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868"
Sep 18 21:54:25.583735 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:54:25.583717    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868\": container with ID starting with 656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868 not found: ID does not exist" containerID="656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868"
Sep 18 21:54:25.583778 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.583744    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868"} err="failed to get container status \"656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868\": rpc error: code = NotFound desc = could not find container \"656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868\": container with ID starting with 656c59356f458a6de9a0f98762f9365bf2d710ea8ae237e45d96990933311868 not found: ID does not exist"
Sep 18 21:54:25.583778 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.583762    2564 scope.go:117] "RemoveContainer" containerID="ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28"
Sep 18 21:54:25.584010 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:54:25.583993    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28\": container with ID starting with ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28 not found: ID does not exist" containerID="ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28"
Sep 18 21:54:25.584048 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.584016    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28"} err="failed to get container status \"ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28\": rpc error: code = NotFound desc = could not find container \"ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28\": container with ID starting with ec8d89283ed532c56f6bb3150ed26809b586c2eac8b4cc1966a12c1a7a83bd28 not found: ID does not exist"
Sep 18 21:54:25.584048 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.584030    2564 scope.go:117] "RemoveContainer" containerID="7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa"
Sep 18 21:54:25.584255 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:54:25.584240    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa\": container with ID starting with 7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa not found: ID does not exist" containerID="7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa"
Sep 18 21:54:25.584401 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.584261    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa"} err="failed to get container status \"7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa\": rpc error: code = NotFound desc = could not find container \"7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa\": container with ID starting with 7e5a1cb7abba1eaf32a8c05b22612dfe45c4b84eb24673600b54ca6f64372caa not found: ID does not exist"
Sep 18 21:54:25.584401 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.584289    2564 scope.go:117] "RemoveContainer" containerID="212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78"
Sep 18 21:54:25.584584 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:54:25.584552    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78\": container with ID starting with 212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78 not found: ID does not exist" containerID="212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78"
Sep 18 21:54:25.584584 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.584573    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78"} err="failed to get container status \"212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78\": rpc error: code = NotFound desc = could not find container \"212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78\": container with ID starting with 212c536e0c3ac2fe40c31f3b910f05508c68dba105ff4139c12fc8d10f3ddd78 not found: ID does not exist"
Sep 18 21:54:25.584828 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.584587    2564 scope.go:117] "RemoveContainer" containerID="751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b"
Sep 18 21:54:25.585428 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:54:25.585351    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b\": container with ID starting with 751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b not found: ID does not exist" containerID="751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b"
Sep 18 21:54:25.585428 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.585381    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b"} err="failed to get container status \"751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b\": rpc error: code = NotFound desc = could not find container \"751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b\": container with ID starting with 751baa69a0e994fc96f8fed5a23bc1d2e025eae49924acbe2f180fa0245a2f1b not found: ID does not exist"
Sep 18 21:54:25.585428 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.585415    2564 scope.go:117] "RemoveContainer" containerID="d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0"
Sep 18 21:54:25.585856 ip-10-0-130-79 kubenswrapper[2564]: E0918 21:54:25.585835    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0\": container with ID starting with d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0 not found: ID does not exist" containerID="d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0"
Sep 18 21:54:25.585928 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.585863    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0"} err="failed to get container status \"d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0\": rpc error: code = NotFound desc = could not find container \"d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0\": container with ID starting with d45e29ff0e5df9348e5d8bebbf676473c02b95f5e3a27fdd6fcd68dd6f5db8e0 not found: ID does not exist"
Sep 18 21:54:25.586678 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.586661    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:54:25.586956 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.586945    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" containerName="registry"
Sep 18 21:54:25.586997 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.586958    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" containerName="registry"
Sep 18 21:54:25.586997 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.586967    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="prom-label-proxy"
Sep 18 21:54:25.586997 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.586974    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="prom-label-proxy"
Sep 18 21:54:25.586997 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.586983    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="init-config-reloader"
Sep 18 21:54:25.586997 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.586989    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="init-config-reloader"
Sep 18 21:54:25.586997 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.586995    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy-web"
Sep 18 21:54:25.586997 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587000    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy-web"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587007    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy-metric"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587012    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy-metric"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587020    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="config-reloader"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587025    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="config-reloader"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587034    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587043    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587052    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="alertmanager"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587061    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="alertmanager"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587115    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec02faa2-62b4-4c66-9ffd-6b24ae4d9259" containerName="registry"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587126    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy-metric"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587136    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="alertmanager"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587145    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="prom-label-proxy"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587154    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="config-reloader"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587163    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy-web"
Sep 18 21:54:25.587283 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.587173    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" containerName="kube-rbac-proxy"
Sep 18 21:54:25.593893 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.593868    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.596346 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.596323    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 21:54:25.597052 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.597031    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 21:54:25.597165 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.597061    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 21:54:25.597255 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.597237    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 21:54:25.597342 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.597259    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 21:54:25.597342 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.597328    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 21:54:25.597445 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.597402    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 21:54:25.597628 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.597612    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 21:54:25.597696 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.597675    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pbqfr\""
Sep 18 21:54:25.604190 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.604169    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 21:54:25.609428 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.609399    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:54:25.619378 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.619351    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb" path="/var/lib/kubelet/pods/7b78e70f-e99f-4c91-9c94-1bd0d82a7ccb/volumes"
Sep 18 21:54:25.734478 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734441    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e7fe62f6-9711-431b-8167-ba977c1f9b75-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734478 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e7fe62f6-9711-431b-8167-ba977c1f9b75-config-out\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734683 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734509    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-web-config\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734683 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734683 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734683 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734653    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5bb\" (UniqueName: \"kubernetes.io/projected/e7fe62f6-9711-431b-8167-ba977c1f9b75-kube-api-access-bs5bb\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734683 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-config-volume\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734835 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734696    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734835 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734720    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734835 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7fe62f6-9711-431b-8167-ba977c1f9b75-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734835 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734793    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e7fe62f6-9711-431b-8167-ba977c1f9b75-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734835 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.734987 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.734838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7fe62f6-9711-431b-8167-ba977c1f9b75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836116 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e7fe62f6-9711-431b-8167-ba977c1f9b75-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836116 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836115    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e7fe62f6-9711-431b-8167-ba977c1f9b75-config-out\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836338 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-web-config\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836338 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836338 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836315    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5bb\" (UniqueName: \"kubernetes.io/projected/e7fe62f6-9711-431b-8167-ba977c1f9b75-kube-api-access-bs5bb\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-config-volume\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836431 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836566 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836448    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836566 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7fe62f6-9711-431b-8167-ba977c1f9b75-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836566 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e7fe62f6-9711-431b-8167-ba977c1f9b75-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836566 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.836738 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.836577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7fe62f6-9711-431b-8167-ba977c1f9b75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.837455 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.837431    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7fe62f6-9711-431b-8167-ba977c1f9b75-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.837606 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.837579    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7fe62f6-9711-431b-8167-ba977c1f9b75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.838175 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.837846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e7fe62f6-9711-431b-8167-ba977c1f9b75-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.839008 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.838978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e7fe62f6-9711-431b-8167-ba977c1f9b75-config-out\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.839105 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.839057    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e7fe62f6-9711-431b-8167-ba977c1f9b75-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.839321 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.839291    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-web-config\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.839498 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.839475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.839577 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.839534    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-config-volume\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.839737 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.839716    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.840001 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.839985    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.840448 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.840430    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.841013 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.840998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7fe62f6-9711-431b-8167-ba977c1f9b75-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.846230 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.846212    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5bb\" (UniqueName: \"kubernetes.io/projected/e7fe62f6-9711-431b-8167-ba977c1f9b75-kube-api-access-bs5bb\") pod \"alertmanager-main-0\" (UID: \"e7fe62f6-9711-431b-8167-ba977c1f9b75\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:25.904314 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:25.904285    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:54:26.184156 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:26.184082    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:54:26.187377 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:54:26.187352    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fe62f6_9711_431b_8167_ba977c1f9b75.slice/crio-56658ae18d38497a1b42e85aeff92a662975da680f91f5b7c0a0c674d326a68b WatchSource:0}: Error finding container 56658ae18d38497a1b42e85aeff92a662975da680f91f5b7c0a0c674d326a68b: Status 404 returned error can't find the container with id 56658ae18d38497a1b42e85aeff92a662975da680f91f5b7c0a0c674d326a68b
Sep 18 21:54:26.528959 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:26.528927    2564 generic.go:356] "Generic (PLEG): container finished" podID="e7fe62f6-9711-431b-8167-ba977c1f9b75" containerID="99ade3f315b1ac994b93d9f9f251229dc885d9893b336a96ade76b58a2a79064" exitCode=0
Sep 18 21:54:26.529133 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:26.529014    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e7fe62f6-9711-431b-8167-ba977c1f9b75","Type":"ContainerDied","Data":"99ade3f315b1ac994b93d9f9f251229dc885d9893b336a96ade76b58a2a79064"}
Sep 18 21:54:26.529133 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:26.529081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e7fe62f6-9711-431b-8167-ba977c1f9b75","Type":"ContainerStarted","Data":"56658ae18d38497a1b42e85aeff92a662975da680f91f5b7c0a0c674d326a68b"}
Sep 18 21:54:27.536765 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:27.536731    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e7fe62f6-9711-431b-8167-ba977c1f9b75","Type":"ContainerStarted","Data":"79a736b5d7618bde2a889660f44d5758e8674726d36d9e6f89a4494c7aaaa252"}
Sep 18 21:54:27.537138 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:27.536815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e7fe62f6-9711-431b-8167-ba977c1f9b75","Type":"ContainerStarted","Data":"0d107f379eae0805639b139f3c6c3d293c5d9ef0f347252931605b2b4145831e"}
Sep 18 21:54:27.537138 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:27.536849    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e7fe62f6-9711-431b-8167-ba977c1f9b75","Type":"ContainerStarted","Data":"fcbd1bf4e575dfb189be6e450e3bba36450dbaf5f2f0fd1ab8afbf8262a5dba9"}
Sep 18 21:54:27.537138 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:27.536876    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e7fe62f6-9711-431b-8167-ba977c1f9b75","Type":"ContainerStarted","Data":"9783b1222425bbd59ffccc1bb384e847b19585c4df6401a2404c079a935a4d92"}
Sep 18 21:54:27.537138 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:27.536898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e7fe62f6-9711-431b-8167-ba977c1f9b75","Type":"ContainerStarted","Data":"5b141ae6c05fcba4bc41f91e3c6e08162d9ad074c4edb704bf811e77ba3ef0ba"}
Sep 18 21:54:27.537138 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:27.536922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e7fe62f6-9711-431b-8167-ba977c1f9b75","Type":"ContainerStarted","Data":"e277da2be0708367e8099a6733d77c2414e41ec90f961472ca7dd143322eaec8"}
Sep 18 21:54:27.585231 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:27.585074    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.569551722 podStartE2EDuration="2.585055141s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.503419ms" isStatefulPod=false podCreationTimestamp="2026-09-18 21:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:54:27.583543771 +0000 UTC m=+246.440199427" watchObservedRunningTime="2026-09-18 21:54:27.585055141 +0000 UTC m=+246.441710792"
Sep 18 21:54:28.048555 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.048521    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5644b867f7-lxt8l"]
Sep 18 21:54:28.052113 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.052087    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.055939 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.055911    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 21:54:28.056280 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.056244    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-48lhg\""
Sep 18 21:54:28.056513 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.056495    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 21:54:28.058302 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.058281    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 21:54:28.058381 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.058337    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 21:54:28.058717 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.058698    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 21:54:28.064624 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.064605    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 21:54:28.070356 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.070337    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5644b867f7-lxt8l"]
Sep 18 21:54:28.156780 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.156754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th5nn\" (UniqueName: \"kubernetes.io/projected/5ce20b8b-f247-46b2-8319-962864241977-kube-api-access-th5nn\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.157000 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.156983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-secret-telemeter-client\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.157207 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.157191    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ce20b8b-f247-46b2-8319-962864241977-metrics-client-ca\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.157362 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.157349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-telemeter-client-tls\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.157514 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.157488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce20b8b-f247-46b2-8319-962864241977-serving-certs-ca-bundle\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.157606 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.157522    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce20b8b-f247-46b2-8319-962864241977-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.157661 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.157610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.157721 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.157685    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-federate-client-tls\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.258065 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.258036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce20b8b-f247-46b2-8319-962864241977-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.258207 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.258089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.258207 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.258108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-federate-client-tls\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.258207 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.258145    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th5nn\" (UniqueName: \"kubernetes.io/projected/5ce20b8b-f247-46b2-8319-962864241977-kube-api-access-th5nn\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.258207 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.258161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-secret-telemeter-client\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.258416 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.258217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ce20b8b-f247-46b2-8319-962864241977-metrics-client-ca\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.258416 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.258260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-telemeter-client-tls\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.258416 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.258322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce20b8b-f247-46b2-8319-962864241977-serving-certs-ca-bundle\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.259072 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.259045    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce20b8b-f247-46b2-8319-962864241977-serving-certs-ca-bundle\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.259072 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.259057    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ce20b8b-f247-46b2-8319-962864241977-metrics-client-ca\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.259284 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.259219    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce20b8b-f247-46b2-8319-962864241977-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.260767 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.260741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-federate-client-tls\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.260859 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.260746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.260947 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.260926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-telemeter-client-tls\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.260984 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.260971    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5ce20b8b-f247-46b2-8319-962864241977-secret-telemeter-client\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.267590 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.267570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th5nn\" (UniqueName: \"kubernetes.io/projected/5ce20b8b-f247-46b2-8319-962864241977-kube-api-access-th5nn\") pod \"telemeter-client-5644b867f7-lxt8l\" (UID: \"5ce20b8b-f247-46b2-8319-962864241977\") " pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.360218 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.360130    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:54:28.360531 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.360495    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="prometheus" containerID="cri-o://c4b4b6417a734d214d7bd74e1b4d02b87e653f140f9f44cc9fa4ac64c941ab46" gracePeriod=600
Sep 18 21:54:28.360698 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.360657    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy" containerID="cri-o://89dcc0ac750d0c4f99ba985b9a2f71f121998ec350ebb668d120b5789f377b90" gracePeriod=600
Sep 18 21:54:28.360698 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.360690    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="thanos-sidecar" containerID="cri-o://e8e6cd321068a7f580f8a176a3a31ad5e0b3744ec7602429a97a0fef4bad4618" gracePeriod=600
Sep 18 21:54:28.360850 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.360787    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4daeef37483dc4ed4b6ec28f1bdd0f823abee88a558bf80bc9ce269caab4ef52" gracePeriod=600
Sep 18 21:54:28.360850 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.360672    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy-web" containerID="cri-o://b14743487f28537a824262a3c1151ed9bf44b6ca2755f3dd0960efa8d32b6fc6" gracePeriod=600
Sep 18 21:54:28.360850 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.360780    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="config-reloader" containerID="cri-o://0de09efb01fa59b22fbbfdbb1ce0add56c937ebb8ed400e407843c14bd8ebde7" gracePeriod=600
Sep 18 21:54:28.363091 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.363059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l"
Sep 18 21:54:28.544555 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544521    2564 generic.go:356] "Generic (PLEG): container finished" podID="84044821-3536-4a4d-8c78-31403988ae43" containerID="4daeef37483dc4ed4b6ec28f1bdd0f823abee88a558bf80bc9ce269caab4ef52" exitCode=0
Sep 18 21:54:28.544555 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544555    2564 generic.go:356] "Generic (PLEG): container finished" podID="84044821-3536-4a4d-8c78-31403988ae43" containerID="89dcc0ac750d0c4f99ba985b9a2f71f121998ec350ebb668d120b5789f377b90" exitCode=0
Sep 18 21:54:28.544934 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544564    2564 generic.go:356] "Generic (PLEG): container finished" podID="84044821-3536-4a4d-8c78-31403988ae43" containerID="e8e6cd321068a7f580f8a176a3a31ad5e0b3744ec7602429a97a0fef4bad4618" exitCode=0
Sep 18 21:54:28.544934 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544576    2564 generic.go:356] "Generic (PLEG): container finished" podID="84044821-3536-4a4d-8c78-31403988ae43" containerID="0de09efb01fa59b22fbbfdbb1ce0add56c937ebb8ed400e407843c14bd8ebde7" exitCode=0
Sep 18 21:54:28.544934 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544584    2564 generic.go:356] "Generic (PLEG): container finished" podID="84044821-3536-4a4d-8c78-31403988ae43" containerID="c4b4b6417a734d214d7bd74e1b4d02b87e653f140f9f44cc9fa4ac64c941ab46" exitCode=0
Sep 18 21:54:28.544934 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerDied","Data":"4daeef37483dc4ed4b6ec28f1bdd0f823abee88a558bf80bc9ce269caab4ef52"}
Sep 18 21:54:28.544934 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544722    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerDied","Data":"89dcc0ac750d0c4f99ba985b9a2f71f121998ec350ebb668d120b5789f377b90"}
Sep 18 21:54:28.544934 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544749    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerDied","Data":"e8e6cd321068a7f580f8a176a3a31ad5e0b3744ec7602429a97a0fef4bad4618"}
Sep 18 21:54:28.544934 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544773    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerDied","Data":"0de09efb01fa59b22fbbfdbb1ce0add56c937ebb8ed400e407843c14bd8ebde7"}
Sep 18 21:54:28.544934 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.544795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerDied","Data":"c4b4b6417a734d214d7bd74e1b4d02b87e653f140f9f44cc9fa4ac64c941ab46"}
Sep 18 21:54:28.675150 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:28.675083    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5644b867f7-lxt8l"]
Sep 18 21:54:28.678903 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:54:28.678864    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce20b8b_f247_46b2_8319_962864241977.slice/crio-6432ee24b49b247e8f40f61fffb6c761863eeeb27b9c4c07fd2539343ccddab2 WatchSource:0}: Error finding container 6432ee24b49b247e8f40f61fffb6c761863eeeb27b9c4c07fd2539343ccddab2: Status 404 returned error can't find the container with id 6432ee24b49b247e8f40f61fffb6c761863eeeb27b9c4c07fd2539343ccddab2
Sep 18 21:54:29.551103 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.551060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l" event={"ID":"5ce20b8b-f247-46b2-8319-962864241977","Type":"ContainerStarted","Data":"6432ee24b49b247e8f40f61fffb6c761863eeeb27b9c4c07fd2539343ccddab2"}
Sep 18 21:54:29.556147 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.556121    2564 generic.go:356] "Generic (PLEG): container finished" podID="84044821-3536-4a4d-8c78-31403988ae43" containerID="b14743487f28537a824262a3c1151ed9bf44b6ca2755f3dd0960efa8d32b6fc6" exitCode=0
Sep 18 21:54:29.556241 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.556170    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerDied","Data":"b14743487f28537a824262a3c1151ed9bf44b6ca2755f3dd0960efa8d32b6fc6"}
Sep 18 21:54:29.613452 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.613423    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:29.773994 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.773960    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-metrics-client-ca\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-tls-assets\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774057    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-config-out\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774083    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774113    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774150    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-tls\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774177    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-1\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774206    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-trusted-ca-bundle\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774232    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-grpc-tls\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774260    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-config\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774304    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-kubelet-serving-ca-bundle\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774333    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-thanos-prometheus-http-client-file\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-metrics-client-certs\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774396    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-web-config\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774398    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774428    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn2g6\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-kube-api-access-rn2g6\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774466    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-0\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774520    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-serving-certs-ca-bundle\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.774547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774568    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-2\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.775741 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774670    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-db\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.775741 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774702    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-kube-rbac-proxy\") pod \"84044821-3536-4a4d-8c78-31403988ae43\" (UID: \"84044821-3536-4a4d-8c78-31403988ae43\") "
Sep 18 21:54:29.775741 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.774988    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-metrics-client-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.775741 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.775027    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:29.776800 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.776414    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:29.776800 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.776639    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:54:29.777331 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.777304    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-config-out" (OuterVolumeSpecName: "config-out") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:54:29.777597 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.777569    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:29.777706 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.777650    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:29.777768 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.777718    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:29.778080 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.777941    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:29.778649 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.778437    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:29.779162 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.778899    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:29.779162 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.779033    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:29.779458 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.779431    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:29.779746 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.779686    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-config" (OuterVolumeSpecName: "config") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:29.780004 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.779972    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:54:29.780150 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.780085    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-kube-api-access-rn2g6" (OuterVolumeSpecName: "kube-api-access-rn2g6") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "kube-api-access-rn2g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:54:29.780363 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.780334    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:29.780568 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.780548    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:29.781141 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.781116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:29.793077 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.793051    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-web-config" (OuterVolumeSpecName: "web-config") pod "84044821-3536-4a4d-8c78-31403988ae43" (UID: "84044821-3536-4a4d-8c78-31403988ae43"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:29.876454 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876411    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-tls-assets\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876454 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876444    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-config-out\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876454 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876460    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876473    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876489    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876501    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876513    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876526    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-grpc-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876539    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876550    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876562    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876574    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-metrics-client-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876588    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-web-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876599    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn2g6\" (UniqueName: \"kubernetes.io/projected/84044821-3536-4a4d-8c78-31403988ae43-kube-api-access-rn2g6\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876612    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876625    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876638    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876651    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84044821-3536-4a4d-8c78-31403988ae43-prometheus-k8s-db\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:29.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:29.876664    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84044821-3536-4a4d-8c78-31403988ae43-secret-kube-rbac-proxy\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:54:30.562889 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.562856    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l" event={"ID":"5ce20b8b-f247-46b2-8319-962864241977","Type":"ContainerStarted","Data":"0cd0679bf20b7522653b951804054d3cd9abaf8e99718f73375e84dffe7ea3e7"}
Sep 18 21:54:30.563319 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.562918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l" event={"ID":"5ce20b8b-f247-46b2-8319-962864241977","Type":"ContainerStarted","Data":"97daa9ebadd7331cd503030392e8350a53c87a38e9e1379ed4d221d9446eb9b4"}
Sep 18 21:54:30.563319 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.562972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l" event={"ID":"5ce20b8b-f247-46b2-8319-962864241977","Type":"ContainerStarted","Data":"e880c2942ae1a72714ea64714aafb0ea067128dce1d65ac60111add0698cc154"}
Sep 18 21:54:30.566124 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.566094    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84044821-3536-4a4d-8c78-31403988ae43","Type":"ContainerDied","Data":"f2211178f93299196b2efdf3078213e39cdd0f14d9be1501c8cb33c54c22a4fa"}
Sep 18 21:54:30.566232 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.566148    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.566232 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.566170    2564 scope.go:117] "RemoveContainer" containerID="4daeef37483dc4ed4b6ec28f1bdd0f823abee88a558bf80bc9ce269caab4ef52"
Sep 18 21:54:30.574874 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.574859    2564 scope.go:117] "RemoveContainer" containerID="89dcc0ac750d0c4f99ba985b9a2f71f121998ec350ebb668d120b5789f377b90"
Sep 18 21:54:30.581758 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.581742    2564 scope.go:117] "RemoveContainer" containerID="b14743487f28537a824262a3c1151ed9bf44b6ca2755f3dd0960efa8d32b6fc6"
Sep 18 21:54:30.588023 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.588005    2564 scope.go:117] "RemoveContainer" containerID="e8e6cd321068a7f580f8a176a3a31ad5e0b3744ec7602429a97a0fef4bad4618"
Sep 18 21:54:30.592719 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.592666    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5644b867f7-lxt8l" podStartSLOduration=0.98979856 podStartE2EDuration="2.592647339s" totalImagesPullingTime="1.602848779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:54:30.591353025 +0000 UTC m=+249.448008669" watchObservedRunningTime="2026-09-18 21:54:30.592647339 +0000 UTC m=+249.449302993"
Sep 18 21:54:30.595401 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.595387    2564 scope.go:117] "RemoveContainer" containerID="0de09efb01fa59b22fbbfdbb1ce0add56c937ebb8ed400e407843c14bd8ebde7"
Sep 18 21:54:30.601864 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.601850    2564 scope.go:117] "RemoveContainer" containerID="c4b4b6417a734d214d7bd74e1b4d02b87e653f140f9f44cc9fa4ac64c941ab46"
Sep 18 21:54:30.608705 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.608691    2564 scope.go:117] "RemoveContainer" containerID="aba12d36043cc6a83165881003ff430f7681662d2e3856affbc6514f5adcafc2"
Sep 18 21:54:30.612331 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.612310    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:54:30.620795 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.620774    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:54:30.659824 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.659799    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:54:30.660114 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660098    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy-web"
Sep 18 21:54:30.660159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660118    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy-web"
Sep 18 21:54:30.660159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660127    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="thanos-sidecar"
Sep 18 21:54:30.660159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660133    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="thanos-sidecar"
Sep 18 21:54:30.660159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660141    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="prometheus"
Sep 18 21:54:30.660159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660147    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="prometheus"
Sep 18 21:54:30.660159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660153    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy-thanos"
Sep 18 21:54:30.660159 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660159    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy-thanos"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660167    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="init-config-reloader"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660173    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="init-config-reloader"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660181    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="config-reloader"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660189    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="config-reloader"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660200    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660206    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660255    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy-web"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660277    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="thanos-sidecar"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660287    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="prometheus"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660295    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy-thanos"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660300    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="kube-rbac-proxy"
Sep 18 21:54:30.660433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.660305    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84044821-3536-4a4d-8c78-31403988ae43" containerName="config-reloader"
Sep 18 21:54:30.668669 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.668647    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.673064 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673046    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-59epskovr0sfc\""
Sep 18 21:54:30.673247 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673219    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 21:54:30.673247 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673231    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mnm42\""
Sep 18 21:54:30.673247 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673234    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 21:54:30.673433 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673335    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 21:54:30.673512 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673496    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 21:54:30.673809 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673745    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 21:54:30.673809 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673792    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 21:54:30.673809 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673804    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 21:54:30.674030 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673821    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 21:54:30.674030 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673852    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 21:54:30.674030 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673870    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 21:54:30.674030 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673804    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 21:54:30.674030 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.673974    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 21:54:30.678022 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.677626    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 21:54:30.681539 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.681521    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 21:54:30.685054 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.685032    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:54:30.784057 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784031    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-config\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784206 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784206 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784206 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784114    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvfs\" (UniqueName: \"kubernetes.io/projected/28a7731a-fd85-47b7-b7c9-982264658a58-kube-api-access-6kvfs\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784206 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784137    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784211    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-web-config\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784331    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784382 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784357    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784538 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784538 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784411    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784538 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784538 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784467    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784538 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784538 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28a7731a-fd85-47b7-b7c9-982264658a58-config-out\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784720 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784720 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28a7731a-fd85-47b7-b7c9-982264658a58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784720 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.784720 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.784612    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.800680 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.800656    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:30.885899 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.885846    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.885899 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.885873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.885899 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.885892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-web-config\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.886103 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.885911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.886103 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.885932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.886103 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.885953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.886103 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.885976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.886484 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.886247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.886651 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.886630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.886770 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.886754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.886899 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.886883    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28a7731a-fd85-47b7-b7c9-982264658a58-config-out\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.887002 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.886987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.887125 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.887109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28a7731a-fd85-47b7-b7c9-982264658a58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.887251 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.887236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888179 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.887629    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888179 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.887696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-config\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888179 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.887725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888179 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.887755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888179 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.887781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvfs\" (UniqueName: \"kubernetes.io/projected/28a7731a-fd85-47b7-b7c9-982264658a58-kube-api-access-6kvfs\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888179 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.887812    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888179 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.888023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888179 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.886881    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888179 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.886581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888767 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.888740    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888840 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.886988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.888896 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.888849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.889429 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.887585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.889429 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.886671    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.889812 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.889786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.889887 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.889838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-web-config\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.890719 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.890696    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.891348 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.891093    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.891788 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.891482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28a7731a-fd85-47b7-b7c9-982264658a58-config-out\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.891788 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.891633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.891788 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.891636    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.891788 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.891632    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.891788 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.891706    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28a7731a-fd85-47b7-b7c9-982264658a58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.892082 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.891906    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28a7731a-fd85-47b7-b7c9-982264658a58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.892792 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.892773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/28a7731a-fd85-47b7-b7c9-982264658a58-config\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.898450 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.898427    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:30.898854 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.898837    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvfs\" (UniqueName: \"kubernetes.io/projected/28a7731a-fd85-47b7-b7c9-982264658a58-kube-api-access-6kvfs\") pod \"prometheus-k8s-0\" (UID: \"28a7731a-fd85-47b7-b7c9-982264658a58\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:30.972627 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.972602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:30.979526 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:30.979502    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:31.285356 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:31.285327    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:54:31.287156 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:54:31.287129    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a7731a_fd85_47b7_b7c9_982264658a58.slice/crio-f1d50549eba5af2990472dc1108b7d311b1527bfbd182c313f5c8f2ab7ce2392 WatchSource:0}: Error finding container f1d50549eba5af2990472dc1108b7d311b1527bfbd182c313f5c8f2ab7ce2392: Status 404 returned error can't find the container with id f1d50549eba5af2990472dc1108b7d311b1527bfbd182c313f5c8f2ab7ce2392
Sep 18 21:54:31.489760 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:31.489731    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:31.564685 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:31.564594    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:31.572633 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:31.572605    2564 generic.go:356] "Generic (PLEG): container finished" podID="28a7731a-fd85-47b7-b7c9-982264658a58" containerID="779abbd7da74d372094fe4fd522fa50a8c22120ca35d601426e21ccd7e72b978" exitCode=0
Sep 18 21:54:31.572775 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:31.572695    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28a7731a-fd85-47b7-b7c9-982264658a58","Type":"ContainerDied","Data":"779abbd7da74d372094fe4fd522fa50a8c22120ca35d601426e21ccd7e72b978"}
Sep 18 21:54:31.572775 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:31.572772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28a7731a-fd85-47b7-b7c9-982264658a58","Type":"ContainerStarted","Data":"f1d50549eba5af2990472dc1108b7d311b1527bfbd182c313f5c8f2ab7ce2392"}
Sep 18 21:54:31.618565 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:31.618543    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84044821-3536-4a4d-8c78-31403988ae43" path="/var/lib/kubelet/pods/84044821-3536-4a4d-8c78-31403988ae43/volumes"
Sep 18 21:54:32.581225 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:32.581199    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 21:54:32.581604 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:32.581575    2564 generic.go:356] "Generic (PLEG): container finished" podID="28a7731a-fd85-47b7-b7c9-982264658a58" containerID="34bff46249ab7b46c9ba558e9ff25346d4c8ccd0674b33d228b424e769a3d880" exitCode=2
Sep 18 21:54:32.581645 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:32.581615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28a7731a-fd85-47b7-b7c9-982264658a58","Type":"ContainerStarted","Data":"73d00516c6b45970f937d8357ce124c752000d17868be926f761ab7d1311307b"}
Sep 18 21:54:32.581698 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:32.581686    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28a7731a-fd85-47b7-b7c9-982264658a58","Type":"ContainerStarted","Data":"03280c6fbe1823fa89895c200dad125269961919890ff16f7a882561cb90d746"}
Sep 18 21:54:32.581733 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:32.581716    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28a7731a-fd85-47b7-b7c9-982264658a58","Type":"ContainerStarted","Data":"6fef3d68e093e2658ae96c770216d953c1ce638cffd0bd8cd88e3b20a74eb97a"}
Sep 18 21:54:32.581769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:32.581739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28a7731a-fd85-47b7-b7c9-982264658a58","Type":"ContainerStarted","Data":"81d3ab9f95f064c36906423ed6b33409c3e226de1f80fe04c428eea0536e0671"}
Sep 18 21:54:32.581769 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:32.581764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28a7731a-fd85-47b7-b7c9-982264658a58","Type":"ContainerStarted","Data":"c3cdbd16633febc04efdef31f90df313d5eb7b53364877ffb3a275a26504f1c7"}
Sep 18 21:54:32.581832 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:32.581788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28a7731a-fd85-47b7-b7c9-982264658a58","Type":"ContainerDied","Data":"34bff46249ab7b46c9ba558e9ff25346d4c8ccd0674b33d228b424e769a3d880"}
Sep 18 21:54:32.582046 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:32.582031    2564 scope.go:117] "RemoveContainer" containerID="34bff46249ab7b46c9ba558e9ff25346d4c8ccd0674b33d228b424e769a3d880"
Sep 18 21:54:33.512163 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:33.512126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:54:33.514387 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:33.514363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba-metrics-certs\") pod \"network-metrics-daemon-vqfxz\" (UID: \"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba\") " pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:54:33.589683 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:33.589660    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 21:54:33.590091 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:33.590069    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"28a7731a-fd85-47b7-b7c9-982264658a58","Type":"ContainerStarted","Data":"1752d0ccdfd68cf49504bb45930b15650d443439afe419b55abebaa358d9dc52"}
Sep 18 21:54:33.624111 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:33.624054    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.6028872339999998 podStartE2EDuration="3.624037061s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.149827ms" isStatefulPod=false podCreationTimestamp="2026-09-18 21:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:54:33.62276508 +0000 UTC m=+252.479420732" watchObservedRunningTime="2026-09-18 21:54:33.624037061 +0000 UTC m=+252.480692713"
Sep 18 21:54:33.717372 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:33.717352    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgp2x\""
Sep 18 21:54:33.724834 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:33.724820    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqfxz"
Sep 18 21:54:34.008329 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:34.008300    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vqfxz"]
Sep 18 21:54:34.013158 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:54:34.013131    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc298ba2e_98fb_4e5c_ab2b_e910ef74e0ba.slice/crio-176bf49896d08314ebc7298cb048c44343f0cf7327b1f4cad01f9c1ef8727fd8 WatchSource:0}: Error finding container 176bf49896d08314ebc7298cb048c44343f0cf7327b1f4cad01f9c1ef8727fd8: Status 404 returned error can't find the container with id 176bf49896d08314ebc7298cb048c44343f0cf7327b1f4cad01f9c1ef8727fd8
Sep 18 21:54:34.596213 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:34.596174    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqfxz" event={"ID":"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba","Type":"ContainerStarted","Data":"176bf49896d08314ebc7298cb048c44343f0cf7327b1f4cad01f9c1ef8727fd8"}
Sep 18 21:54:35.601892 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:35.601853    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqfxz" event={"ID":"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba","Type":"ContainerStarted","Data":"15689d1e4d95efe9890c6db80d8074890d1033abc46b2e7d23e7825a9b316805"}
Sep 18 21:54:35.602376 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:35.601917    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqfxz" event={"ID":"c298ba2e-98fb-4e5c-ab2b-e910ef74e0ba","Type":"ContainerStarted","Data":"fe171f59b41ad6503b30b5eb53ca74023d7e4d942e62beba7e2a50aa6bda9670"}
Sep 18 21:54:35.621358 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:35.621316    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vqfxz" podStartSLOduration=251.653105822 podStartE2EDuration="4m12.621299121s" totalImagesPullingTime="968.193299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:54:35.619922198 +0000 UTC m=+254.476577850" watchObservedRunningTime="2026-09-18 21:54:35.621299121 +0000 UTC m=+254.477954773"
Sep 18 21:54:35.739130 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:35.739101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:35.844088 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:35.844053    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:35.979824 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:35.979764    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:54:41.860296 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:41.860242    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:41.961220 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:41.961194    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:42.020789 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:42.020766    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:54:44.963654 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:54:44.963575    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:55:21.553332 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:21.553301    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 21:55:21.561318 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:21.561258    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 21:55:21.574568 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:21.574548    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 21:55:26.858403 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:26.858365    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:55:30.979988 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:30.979938    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:55:30.994981 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:30.994960    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:55:31.835337 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:31.835311    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:55:38.149583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:38.149549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:55:38.219711 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:38.219679    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:55:48.815120 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:48.815083    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:55:48.898127 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:48.898100    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:55:50.111155 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:50.111119    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:55:50.180167 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:55:50.180139    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:56:00.187825 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:56:00.187783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:56:00.282188 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:56:00.282161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:56:00.384602 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:56:00.384574    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:56:06.359622 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:56:06.359587    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:56:06.457572 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:56:06.457539    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:56:06.554564 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:56:06.554511    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:56:49.738203 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:56:49.738163    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:56:51.096752 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:56:51.096714    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:04.339546 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:04.339500    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:04.417587 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:04.417538    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:13.240920 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:13.240885    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:13.364414 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:13.364382    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:20.400295 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:20.400231    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:20.782938 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:20.782901    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:21.195898 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:21.195802    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:21.340955 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:21.340911    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:21.664799 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:21.664610    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:21.809707 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:21.809663    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:22.269831 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:22.269798    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:22.463574 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:22.463538    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:22.626155 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:22.626117    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:57:22.778437 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:57:22.778403    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:11.434583 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:11.434542    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:11.545697 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:11.545666    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:23.075785 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:23.075748    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:23.389608 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:23.389511    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:25.532356 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:25.532313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:25.683307 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:25.683248    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:26.558722 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:26.558667    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:27.708155 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:27.708118    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:28.081068 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:28.081018    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:30.271586 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:30.271556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:30.454036 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:30.453997    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:31.353757 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.353717    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vrtd2"]
Sep 18 21:58:31.360031 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.358012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:31.362556 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.362535    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 21:58:31.363112 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.363092    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-65slb\""
Sep 18 21:58:31.363191 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.363107    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 21:58:31.396011 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.395986    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vrtd2"]
Sep 18 21:58:31.455473 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.455436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c82q\" (UniqueName: \"kubernetes.io/projected/1a86c0b2-5bc6-48cf-a5f7-84743bae85a0-kube-api-access-4c82q\") pod \"cert-manager-webhook-6486ffd997-vrtd2\" (UID: \"1a86c0b2-5bc6-48cf-a5f7-84743bae85a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:31.455668 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.455565    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a86c0b2-5bc6-48cf-a5f7-84743bae85a0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vrtd2\" (UID: \"1a86c0b2-5bc6-48cf-a5f7-84743bae85a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:31.557041 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.557009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a86c0b2-5bc6-48cf-a5f7-84743bae85a0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vrtd2\" (UID: \"1a86c0b2-5bc6-48cf-a5f7-84743bae85a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:31.557229 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.557060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c82q\" (UniqueName: \"kubernetes.io/projected/1a86c0b2-5bc6-48cf-a5f7-84743bae85a0-kube-api-access-4c82q\") pod \"cert-manager-webhook-6486ffd997-vrtd2\" (UID: \"1a86c0b2-5bc6-48cf-a5f7-84743bae85a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:31.590225 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.590192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c82q\" (UniqueName: \"kubernetes.io/projected/1a86c0b2-5bc6-48cf-a5f7-84743bae85a0-kube-api-access-4c82q\") pod \"cert-manager-webhook-6486ffd997-vrtd2\" (UID: \"1a86c0b2-5bc6-48cf-a5f7-84743bae85a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:31.590225 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.590213    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a86c0b2-5bc6-48cf-a5f7-84743bae85a0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vrtd2\" (UID: \"1a86c0b2-5bc6-48cf-a5f7-84743bae85a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:31.685401 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.685315    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:31.965547 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.965523    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vrtd2"]
Sep 18 21:58:31.968342 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:58:31.968311    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a86c0b2_5bc6_48cf_a5f7_84743bae85a0.slice/crio-b10a73c5ae006b553e9bc7a8165fac39ccc105d25897325095507c84f1379a21 WatchSource:0}: Error finding container b10a73c5ae006b553e9bc7a8165fac39ccc105d25897325095507c84f1379a21: Status 404 returned error can't find the container with id b10a73c5ae006b553e9bc7a8165fac39ccc105d25897325095507c84f1379a21
Sep 18 21:58:31.970996 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:31.970976    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:58:32.545104 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:32.545070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2" event={"ID":"1a86c0b2-5bc6-48cf-a5f7-84743bae85a0","Type":"ContainerStarted","Data":"b10a73c5ae006b553e9bc7a8165fac39ccc105d25897325095507c84f1379a21"}
Sep 18 21:58:32.713944 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:32.713905    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:33.006711 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:33.006629    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:33.128921 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:33.128880    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:33.286493 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:33.286448    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:33.577187 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:33.577109    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:33.732794 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:33.732755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:36.564798 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:36.564765    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2" event={"ID":"1a86c0b2-5bc6-48cf-a5f7-84743bae85a0","Type":"ContainerStarted","Data":"7e105272c929c44b7fdfdaeaa87b5a095047893de7f344362f10937541b15025"}
Sep 18 21:58:36.565227 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:36.564924    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:36.611684 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:36.611634    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2" podStartSLOduration=1.948932284 podStartE2EDuration="5.611617662s" totalImagesPullingTime="3.662685378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:58:36.610431878 +0000 UTC m=+495.467087531" watchObservedRunningTime="2026-09-18 21:58:36.611617662 +0000 UTC m=+495.468273313"
Sep 18 21:58:37.019141 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:37.019023    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:37.242586 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:37.242551    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:37.806622 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:37.806590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:37.943521 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:37.943490    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:39.871176 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:39.871140    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:41.036205 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.036170    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bqsxg"]
Sep 18 21:58:41.039552 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.039536    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bqsxg"
Sep 18 21:58:41.048448 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.048433    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-whtc2\""
Sep 18 21:58:41.058003 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.057974    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bqsxg"]
Sep 18 21:58:41.142056 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.142020    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mgt\" (UniqueName: \"kubernetes.io/projected/a299eac8-2156-461e-8846-c949999dda1d-kube-api-access-r7mgt\") pod \"cert-manager-55b4d954d5-bqsxg\" (UID: \"a299eac8-2156-461e-8846-c949999dda1d\") " pod="cert-manager/cert-manager-55b4d954d5-bqsxg"
Sep 18 21:58:41.142056 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.142062    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a299eac8-2156-461e-8846-c949999dda1d-bound-sa-token\") pod \"cert-manager-55b4d954d5-bqsxg\" (UID: \"a299eac8-2156-461e-8846-c949999dda1d\") " pod="cert-manager/cert-manager-55b4d954d5-bqsxg"
Sep 18 21:58:41.243523 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.243491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a299eac8-2156-461e-8846-c949999dda1d-bound-sa-token\") pod \"cert-manager-55b4d954d5-bqsxg\" (UID: \"a299eac8-2156-461e-8846-c949999dda1d\") " pod="cert-manager/cert-manager-55b4d954d5-bqsxg"
Sep 18 21:58:41.243695 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.243584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mgt\" (UniqueName: \"kubernetes.io/projected/a299eac8-2156-461e-8846-c949999dda1d-kube-api-access-r7mgt\") pod \"cert-manager-55b4d954d5-bqsxg\" (UID: \"a299eac8-2156-461e-8846-c949999dda1d\") " pod="cert-manager/cert-manager-55b4d954d5-bqsxg"
Sep 18 21:58:41.258424 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.258395    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a299eac8-2156-461e-8846-c949999dda1d-bound-sa-token\") pod \"cert-manager-55b4d954d5-bqsxg\" (UID: \"a299eac8-2156-461e-8846-c949999dda1d\") " pod="cert-manager/cert-manager-55b4d954d5-bqsxg"
Sep 18 21:58:41.259999 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.259981    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mgt\" (UniqueName: \"kubernetes.io/projected/a299eac8-2156-461e-8846-c949999dda1d-kube-api-access-r7mgt\") pod \"cert-manager-55b4d954d5-bqsxg\" (UID: \"a299eac8-2156-461e-8846-c949999dda1d\") " pod="cert-manager/cert-manager-55b4d954d5-bqsxg"
Sep 18 21:58:41.348110 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.348022    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bqsxg"
Sep 18 21:58:41.638911 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:41.638826    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bqsxg"]
Sep 18 21:58:42.572011 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:42.571976    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vrtd2"
Sep 18 21:58:42.590645 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:42.590611    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bqsxg" event={"ID":"a299eac8-2156-461e-8846-c949999dda1d","Type":"ContainerStarted","Data":"a35cdfeca10eefdccc83531bc7f76828d11a50b6f785f62f31e1ca9b15de4061"}
Sep 18 21:58:42.590645 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:42.590649    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bqsxg" event={"ID":"a299eac8-2156-461e-8846-c949999dda1d","Type":"ContainerStarted","Data":"a0729f6dd88f47d40ae9d485b541a1566b12063f8910f71c3e8d0e69c15f532f"}
Sep 18 21:58:42.640674 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:42.640626    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bqsxg" podStartSLOduration=1.64060923 podStartE2EDuration="1.64060923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:58:42.640443849 +0000 UTC m=+501.497099504" watchObservedRunningTime="2026-09-18 21:58:42.64060923 +0000 UTC m=+501.497264881"
Sep 18 21:58:45.092492 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:45.092452    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:45.235634 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:45.235591    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:56.677702 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:56.677657    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:58:57.018090 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:58:57.018000    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:02.285822 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:02.285783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:02.378773 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:02.378739    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:04.168510 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:04.168474    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:04.961830 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:04.961792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:05.114554 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.114519    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"]
Sep 18 21:59:05.117680 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.117664    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.121420 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.121399    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 21:59:05.122044 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.122028    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:59:05.122342 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.122310    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 21:59:05.122449 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.122345    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 21:59:05.122546 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.122529    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 21:59:05.125829 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.125808    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b6kjd\""
Sep 18 21:59:05.142123 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.142099    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"]
Sep 18 21:59:05.159685 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.159655    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:05.252887 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.252805    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-metrics-cert\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.252887 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.252858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-cert\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.252887 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.252877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4cd\" (UniqueName: \"kubernetes.io/projected/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-kube-api-access-mj4cd\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.253337 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.252931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-manager-config\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.354257 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.354223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-manager-config\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.354440 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.354326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-metrics-cert\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.354440 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.354388    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-cert\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.354440 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.354405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4cd\" (UniqueName: \"kubernetes.io/projected/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-kube-api-access-mj4cd\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.354927 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.354903    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-manager-config\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.356747 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.356731    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-cert\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.356877 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.356862    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-metrics-cert\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.363103 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.363081    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4cd\" (UniqueName: \"kubernetes.io/projected/4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f-kube-api-access-mj4cd\") pod \"lws-controller-manager-7fdf8f455-wdj86\" (UID: \"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.427288 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.427234    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:05.704389 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:05.704229    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"]
Sep 18 21:59:05.707109 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:59:05.707083    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5ea3c8_b9e6_43cb_a3a9_670e5d2a0b0f.slice/crio-45a16fb5d8cd7e1f741c5561966b85e131edf420714274dd49a28c459c35c302 WatchSource:0}: Error finding container 45a16fb5d8cd7e1f741c5561966b85e131edf420714274dd49a28c459c35c302: Status 404 returned error can't find the container with id 45a16fb5d8cd7e1f741c5561966b85e131edf420714274dd49a28c459c35c302
Sep 18 21:59:06.687755 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:06.687715    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86" event={"ID":"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f","Type":"ContainerStarted","Data":"45a16fb5d8cd7e1f741c5561966b85e131edf420714274dd49a28c459c35c302"}
Sep 18 21:59:06.742748 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:06.742715    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:06.909187 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:06.909150    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:09.701669 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:09.701630    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86" event={"ID":"4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f","Type":"ContainerStarted","Data":"2be03ee4b8db31b4bf3d327c6078ca7a2e99854599d5af52c4737f2a360c0d5b"}
Sep 18 21:59:09.702116 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:09.701889    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:09.721768 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:09.721724    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86" podStartSLOduration=1.353521059 podStartE2EDuration="4.721709644s" totalImagesPullingTime="3.368188585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:59:09.720397933 +0000 UTC m=+528.577053584" watchObservedRunningTime="2026-09-18 21:59:09.721709644 +0000 UTC m=+528.578365296"
Sep 18 21:59:10.151081 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.151051    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"]
Sep 18 21:59:10.154566 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.154546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.158410 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.158386    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-fcshg\""
Sep 18 21:59:10.158532 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.158495    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 21:59:10.158593 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.158564    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 21:59:10.159114 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.159096    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 21:59:10.179319 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.179299    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"]
Sep 18 21:59:10.216050 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.216020    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:10.300894 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.300870    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.300983 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.300898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.300983 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.300935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.300983 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.300951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstqc\" (UniqueName: \"kubernetes.io/projected/f1ccfe42-36a8-4352-8ea6-9017406392bf-kube-api-access-sstqc\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.301091 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.301006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.301091 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.301034    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.301091 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.301052    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.301182 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.301093    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ccfe42-36a8-4352-8ea6-9017406392bf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.301182 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.301124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.320856 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.320832    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:10.401503 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401503 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401489    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401600 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401600 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401539    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401725 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstqc\" (UniqueName: \"kubernetes.io/projected/f1ccfe42-36a8-4352-8ea6-9017406392bf-kube-api-access-sstqc\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401725 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401866 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401866 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401986 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401876    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401986 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ccfe42-36a8-4352-8ea6-9017406392bf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.401986 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.401962    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.402177 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.402157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.402223 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.402206    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.402707 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.402686    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ccfe42-36a8-4352-8ea6-9017406392bf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.403737 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.403717    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.403887 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.403870    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.410221 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.410199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ccfe42-36a8-4352-8ea6-9017406392bf-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.410739 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.410709    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstqc\" (UniqueName: \"kubernetes.io/projected/f1ccfe42-36a8-4352-8ea6-9017406392bf-kube-api-access-sstqc\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-s8kvl\" (UID: \"f1ccfe42-36a8-4352-8ea6-9017406392bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.465528 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.465504    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:10.749919 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:10.749805    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"]
Sep 18 21:59:10.752373 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:59:10.752345    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ccfe42_36a8_4352_8ea6_9017406392bf.slice/crio-8bc5db6ac33f959fe1277f0deb74a5ed17913370d4755eda1740be2118ad39fb WatchSource:0}: Error finding container 8bc5db6ac33f959fe1277f0deb74a5ed17913370d4755eda1740be2118ad39fb: Status 404 returned error can't find the container with id 8bc5db6ac33f959fe1277f0deb74a5ed17913370d4755eda1740be2118ad39fb
Sep 18 21:59:11.712919 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:11.712867    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" event={"ID":"f1ccfe42-36a8-4352-8ea6-9017406392bf","Type":"ContainerStarted","Data":"8bc5db6ac33f959fe1277f0deb74a5ed17913370d4755eda1740be2118ad39fb"}
Sep 18 21:59:13.504546 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:13.504509    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:13.747954 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:13.747912    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 21:59:13.748070 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:13.747984    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 21:59:13.748070 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:13.748015    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 21:59:14.727113 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:14.727077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" event={"ID":"f1ccfe42-36a8-4352-8ea6-9017406392bf","Type":"ContainerStarted","Data":"a17c80f94bc83d24f7033b39332fb74255b523765d66e534c7c7e13376cc370d"}
Sep 18 21:59:14.763369 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:14.763323    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" podStartSLOduration=1.7702492410000001 podStartE2EDuration="4.763306574s" totalImagesPullingTime="2.993057333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:59:14.762238533 +0000 UTC m=+533.618894196" watchObservedRunningTime="2026-09-18 21:59:14.763306574 +0000 UTC m=+533.619962226"
Sep 18 21:59:15.465941 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:15.465912    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:15.470161 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:15.470136    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:15.731249 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:15.731171    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:15.732136 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:15.732115    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 21:59:15.969828 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:15.969792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:16.358374 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:16.358335    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:20.708082 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:20.708053    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-wdj86"
Sep 18 21:59:29.112260 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:29.112220    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:34.426763 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:34.426723    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:36.970392 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:36.970356    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:37.369168 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:37.369122    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:38.079022 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:38.078988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:38.217594 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:38.217557    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:38.737410 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:38.737378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:38.915734 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:38.915696    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:39.345218 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:39.345182    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:40.514704 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:40.514666    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:41.030201 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:41.030169    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:41.225385 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:41.225342    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:41.384979 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:41.384891    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:44.859422 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:44.859390    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:45.080117 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:45.080075    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:46.696366 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:46.696329    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"]
Sep 18 21:59:46.705398 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:46.705373    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"
Sep 18 21:59:46.708209 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:46.708186    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 21:59:46.708339 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:46.708186    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 21:59:46.708849 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:46.708835    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gmh87\""
Sep 18 21:59:46.725436 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:46.725402    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"]
Sep 18 21:59:46.816489 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:46.816446    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bvfs\" (UniqueName: \"kubernetes.io/projected/9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4-kube-api-access-5bvfs\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-tbv6j\" (UID: \"9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"
Sep 18 21:59:46.917061 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:46.917022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvfs\" (UniqueName: \"kubernetes.io/projected/9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4-kube-api-access-5bvfs\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-tbv6j\" (UID: \"9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"
Sep 18 21:59:46.925826 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:46.925789    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvfs\" (UniqueName: \"kubernetes.io/projected/9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4-kube-api-access-5bvfs\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-tbv6j\" (UID: \"9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"
Sep 18 21:59:47.015770 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:47.015678    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"
Sep 18 21:59:47.330276 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:47.330234    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"]
Sep 18 21:59:47.334557 ip-10-0-130-79 kubenswrapper[2564]: W0918 21:59:47.334529    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c07c347_e82c_4dc2_b1c7_2ef727ccd7b4.slice/crio-8be32e8104c0f3b6a0e2499e50aab8d584f285e90484ebeb1e07bea6249075a2 WatchSource:0}: Error finding container 8be32e8104c0f3b6a0e2499e50aab8d584f285e90484ebeb1e07bea6249075a2: Status 404 returned error can't find the container with id 8be32e8104c0f3b6a0e2499e50aab8d584f285e90484ebeb1e07bea6249075a2
Sep 18 21:59:47.801772 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:47.801729    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:47.865927 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:47.865892    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j" event={"ID":"9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4","Type":"ContainerStarted","Data":"8be32e8104c0f3b6a0e2499e50aab8d584f285e90484ebeb1e07bea6249075a2"}
Sep 18 21:59:48.102277 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:48.102166    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:53.064321 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:53.064284    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:53.203819 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:53.203783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:53.443379 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:53.443285    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:53.938563 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:53.938535    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:57.508256 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:57.508218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:57.688565 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:57.688507    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:57.874475 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:57.874437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:58.160936 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:58.160850    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 21:59:58.918518 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:58.918482    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j" event={"ID":"9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4","Type":"ContainerStarted","Data":"7aaa160301c8505786fc65f291fcd244a7b2901b8fe8f1e462aa5c5afb0837f4"}
Sep 18 21:59:58.918909 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:58.918567    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"
Sep 18 21:59:58.953521 ip-10-0-130-79 kubenswrapper[2564]: I0918 21:59:58.953470    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j" podStartSLOduration=2.334321265 podStartE2EDuration="12.953454112s" totalImagesPullingTime="10.619132847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:59:58.951369073 +0000 UTC m=+577.808024730" watchObservedRunningTime="2026-09-18 21:59:58.953454112 +0000 UTC m=+577.810109764"
Sep 18 22:00:01.546189 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:01.546139    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:01.904878 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:01.904781    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:05.097041 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:05.096999    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:05.432557 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:05.432462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:07.254374 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:07.254338    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:07.677095 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:07.677058    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:09.925034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:09.925002    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j"
Sep 18 22:00:12.002236 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:12.002194    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:12.681196 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:12.681165    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:13.196626 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:13.196590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:13.512862 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:13.512760    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:13.835721 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:13.835691    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:15.557139 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:15.557098    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:17.554850 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:17.554807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:17.664296 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:17.664240    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:17.765207 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:17.765171    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:21.588303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:21.588256    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:00:21.590787 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:21.590767    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:00:24.598864 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:24.598834    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:24.744279 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:24.744245    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:44.556292 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:44.556186    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:44.660122 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:44.660088    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:44.760621 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:44.760586    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:45.005310 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.005217    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"]
Sep 18 22:00:45.008987 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.008968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:00:45.011553 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.011533    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wmx9d\""
Sep 18 22:00:45.011803 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.011778    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 22:00:45.021971 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.021950    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"]
Sep 18 22:00:45.058849 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.058818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"]
Sep 18 22:00:45.132891 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.132860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbl6\" (UniqueName: \"kubernetes.io/projected/0141d54d-83f6-412f-b003-6c8b30e52d7c-kube-api-access-thbl6\") pod \"limitador-limitador-84994f7ffc-2z8v7\" (UID: \"0141d54d-83f6-412f-b003-6c8b30e52d7c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:00:45.132891 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.132897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0141d54d-83f6-412f-b003-6c8b30e52d7c-config-file\") pod \"limitador-limitador-84994f7ffc-2z8v7\" (UID: \"0141d54d-83f6-412f-b003-6c8b30e52d7c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:00:45.233511 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.233466    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbl6\" (UniqueName: \"kubernetes.io/projected/0141d54d-83f6-412f-b003-6c8b30e52d7c-kube-api-access-thbl6\") pod \"limitador-limitador-84994f7ffc-2z8v7\" (UID: \"0141d54d-83f6-412f-b003-6c8b30e52d7c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:00:45.233683 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.233520    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0141d54d-83f6-412f-b003-6c8b30e52d7c-config-file\") pod \"limitador-limitador-84994f7ffc-2z8v7\" (UID: \"0141d54d-83f6-412f-b003-6c8b30e52d7c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:00:45.234216 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.234198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0141d54d-83f6-412f-b003-6c8b30e52d7c-config-file\") pod \"limitador-limitador-84994f7ffc-2z8v7\" (UID: \"0141d54d-83f6-412f-b003-6c8b30e52d7c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:00:45.242597 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.242576    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbl6\" (UniqueName: \"kubernetes.io/projected/0141d54d-83f6-412f-b003-6c8b30e52d7c-kube-api-access-thbl6\") pod \"limitador-limitador-84994f7ffc-2z8v7\" (UID: \"0141d54d-83f6-412f-b003-6c8b30e52d7c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:00:45.318869 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.318836    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:00:45.420117 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.420079    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:45.529767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.529724    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:00:45.600185 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:45.600129    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"]
Sep 18 22:00:45.602501 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:00:45.602469    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0141d54d_83f6_412f_b003_6c8b30e52d7c.slice/crio-072784b5e78bac8f11f4de063363397882c315929f29d0b8df52fcac4478c838 WatchSource:0}: Error finding container 072784b5e78bac8f11f4de063363397882c315929f29d0b8df52fcac4478c838: Status 404 returned error can't find the container with id 072784b5e78bac8f11f4de063363397882c315929f29d0b8df52fcac4478c838
Sep 18 22:00:46.133437 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:46.133404    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7" event={"ID":"0141d54d-83f6-412f-b003-6c8b30e52d7c","Type":"ContainerStarted","Data":"072784b5e78bac8f11f4de063363397882c315929f29d0b8df52fcac4478c838"}
Sep 18 22:00:47.139691 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:47.139658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7" event={"ID":"0141d54d-83f6-412f-b003-6c8b30e52d7c","Type":"ContainerStarted","Data":"92e31fed0a3a718a9b13f428ba86fecbe259e0dc1d40a3f1ff907182c0bb1a4e"}
Sep 18 22:00:47.140108 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:47.139769    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:00:47.172911 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:47.172852    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7" podStartSLOduration=2.182085357 podStartE2EDuration="3.172833342s" totalImagesPullingTime="990.747985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:47.171783549 +0000 UTC m=+626.028439202" watchObservedRunningTime="2026-09-18 22:00:47.172833342 +0000 UTC m=+626.029488994"
Sep 18 22:00:58.144985 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:00:58.144956    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2z8v7"
Sep 18 22:01:04.366337 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:04.366297    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:04.622896 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:04.622807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:13.540902 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:13.540863    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:13.790287 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:13.790235    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:26.257747 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:26.257710    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:45.447143 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:45.447103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:45.556095 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:45.556057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:45.649889 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:45.649844    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:45.787429 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:45.787394    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:46.037918 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:46.037837    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:46.426443 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:46.426352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:01:46.534579 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:01:46.534543    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:03:05.852627 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:03:05.852592    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:03:07.474944 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:03:07.474910    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:03:07.598905 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:03:07.598867    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:03:19.988758 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:03:19.988711    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:04:30.451717 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:04:30.451683    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:04:42.254252 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:04:42.254194    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:04:42.357192 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:04:42.357130    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:04:55.120338 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:04:55.120302    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:04:55.320583 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:04:55.320541    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:09.658701 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:09.658617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:10.129011 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.128976    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"]
Sep 18 22:05:10.132379 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.132363    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:05:10.137003 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.136981    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 22:05:10.137116 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.136982    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-px4tn\""
Sep 18 22:05:10.137116 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.136981    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 22:05:10.137307 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.137293    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-whrzw\""
Sep 18 22:05:10.142213 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.142199    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 22:05:10.155015 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.154995    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"]
Sep 18 22:05:10.194173 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.194143    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:10.246906 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.246886    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4skh\" (UniqueName: \"kubernetes.io/projected/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-kube-api-access-f4skh\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr\" (UID: \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:05:10.247002 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.246954    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr\" (UID: \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:05:10.347589 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.347564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr\" (UID: \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:05:10.347711 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.347613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4skh\" (UniqueName: \"kubernetes.io/projected/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-kube-api-access-f4skh\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr\" (UID: \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:05:10.349939 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.349922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr\" (UID: \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:05:10.357668 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.357649    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4skh\" (UniqueName: \"kubernetes.io/projected/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-kube-api-access-f4skh\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr\" (UID: \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:05:10.469352 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.469258    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:05:10.740651 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.740626    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"]
Sep 18 22:05:10.743132 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:05:10.743102    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5140ebfe_716f_4ca6_9e29_ac09ee4a34cb.slice/crio-2996f856067d34c0fa785a072e924f4ff6c6145e326c7313b7a80278f340114d WatchSource:0}: Error finding container 2996f856067d34c0fa785a072e924f4ff6c6145e326c7313b7a80278f340114d: Status 404 returned error can't find the container with id 2996f856067d34c0fa785a072e924f4ff6c6145e326c7313b7a80278f340114d
Sep 18 22:05:10.745126 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:10.745108    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:05:11.331328 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:11.331289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" event={"ID":"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb","Type":"ContainerStarted","Data":"2996f856067d34c0fa785a072e924f4ff6c6145e326c7313b7a80278f340114d"}
Sep 18 22:05:13.341385 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:13.341349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" event={"ID":"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb","Type":"ContainerStarted","Data":"81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e"}
Sep 18 22:05:13.341777 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:13.341497    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:05:13.362145 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:13.362084    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" podStartSLOduration=1.6689641160000002 podStartE2EDuration="3.362064374s" totalImagesPullingTime="1.693100258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:05:13.359351808 +0000 UTC m=+892.216007461" watchObservedRunningTime="2026-09-18 22:05:13.362064374 +0000 UTC m=+892.218720027"
Sep 18 22:05:20.381679 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:20.381646    2564 generic.go:356] "Generic (PLEG): container finished" podID="f1ccfe42-36a8-4352-8ea6-9017406392bf" containerID="a17c80f94bc83d24f7033b39332fb74255b523765d66e534c7c7e13376cc370d" exitCode=0
Sep 18 22:05:20.382090 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:20.381718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" event={"ID":"f1ccfe42-36a8-4352-8ea6-9017406392bf","Type":"ContainerDied","Data":"a17c80f94bc83d24f7033b39332fb74255b523765d66e534c7c7e13376cc370d"}
Sep 18 22:05:20.382148 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:20.382133    2564 scope.go:117] "RemoveContainer" containerID="a17c80f94bc83d24f7033b39332fb74255b523765d66e534c7c7e13376cc370d"
Sep 18 22:05:20.383159 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:20.383129    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 22:05:20.383228 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:20.383202    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 22:05:20.383280 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:20.383235    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 22:05:21.389858 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:21.389820    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" event={"ID":"f1ccfe42-36a8-4352-8ea6-9017406392bf","Type":"ContainerStarted","Data":"4f989a26e5f25a6baa9cb0e864e861f4266df3a0053e27163632f54ab03db8a9"}
Sep 18 22:05:21.460129 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:21.460086    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:21.466800 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:21.466763    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 22:05:21.466981 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:21.466900    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 18 22:05:21.466981 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:21.466953    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" podUID="f1ccfe42-36a8-4352-8ea6-9017406392bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 18 22:05:21.624360 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:21.624330    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:05:21.624566 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:21.624377    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:05:21.669063 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:21.668918    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:22.466620 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:22.466586    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 18 22:05:22.467192 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:22.467162    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" podUID="f1ccfe42-36a8-4352-8ea6-9017406392bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 18 22:05:22.799183 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:22.799148    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:23.466077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:23.466030    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 18 22:05:23.466311 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:23.466105    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" podUID="f1ccfe42-36a8-4352-8ea6-9017406392bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 18 22:05:24.466425 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:24.466380    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 18 22:05:24.466918 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:24.466452    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" podUID="f1ccfe42-36a8-4352-8ea6-9017406392bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 18 22:05:25.466491 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:25.466443    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 18 22:05:25.466997 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:25.466526    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" podUID="f1ccfe42-36a8-4352-8ea6-9017406392bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 18 22:05:26.466610 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:26.466580    2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 18 22:05:26.466984 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:26.466645    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl" podUID="f1ccfe42-36a8-4352-8ea6-9017406392bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 18 22:05:27.469962 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:27.469931    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 22:05:27.470400 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:27.470199    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 22:05:27.470873 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:27.470856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl"
Sep 18 22:05:34.938194 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:34.938153    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:44.349970 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:44.349928    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:05:48.938106 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:48.938074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:50.396684 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:50.396645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:53.046798 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:53.046766    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"]
Sep 18 22:05:53.048617 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:53.048589    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:05:53.065868 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:53.065831    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:05:53.165180 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:05:53.165147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:01.362431 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:01.362399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:01.471652 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:01.471612    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:03.048950 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:03.048912    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:08.066982 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:08.066932    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main" containerID="cri-o://81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e" gracePeriod=45
Sep 18 22:06:11.503850 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:11.503815    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:11.601918 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:11.601872    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:12.085433 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:12.085398    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:12.155688 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:12.155638    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:13.048745 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:13.048711    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:23.048846 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:23.048807    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:33.048564 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:33.048525    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:38.300822 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.300799    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:06:38.472469 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.472388    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-tls-certs\") pod \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\" (UID: \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\") "
Sep 18 22:06:38.472611 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.472476    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4skh\" (UniqueName: \"kubernetes.io/projected/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-kube-api-access-f4skh\") pod \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\" (UID: \"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb\") "
Sep 18 22:06:38.474636 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.474606    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" (UID: "5140ebfe-716f-4ca6-9e29-ac09ee4a34cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:06:38.474746 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.474644    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-kube-api-access-f4skh" (OuterVolumeSpecName: "kube-api-access-f4skh") pod "5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" (UID: "5140ebfe-716f-4ca6-9e29-ac09ee4a34cb"). InnerVolumeSpecName "kube-api-access-f4skh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:06:38.574115 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.574088    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:06:38.574115 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.574114    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4skh\" (UniqueName: \"kubernetes.io/projected/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb-kube-api-access-f4skh\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:06:38.757559 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.757496    2564 generic.go:356] "Generic (PLEG): container finished" podID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerID="81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e" exitCode=0
Sep 18 22:06:38.757559 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.757537    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"
Sep 18 22:06:38.757559 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.757551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" event={"ID":"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb","Type":"ContainerDied","Data":"81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e"}
Sep 18 22:06:38.757758 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.757592    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr" event={"ID":"5140ebfe-716f-4ca6-9e29-ac09ee4a34cb","Type":"ContainerDied","Data":"2996f856067d34c0fa785a072e924f4ff6c6145e326c7313b7a80278f340114d"}
Sep 18 22:06:38.757758 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.757612    2564 scope.go:117] "RemoveContainer" containerID="81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e"
Sep 18 22:06:38.766553 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.766537    2564 scope.go:117] "RemoveContainer" containerID="81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e"
Sep 18 22:06:38.766794 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:06:38.766776    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e\": container with ID starting with 81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e not found: ID does not exist" containerID="81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e"
Sep 18 22:06:38.766831 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.766802    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e"} err="failed to get container status \"81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e\": rpc error: code = NotFound desc = could not find container \"81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e\": container with ID starting with 81132426138e3a164e67767ede076745962e26c77ce0e7637a6c2c77eecb2c5e not found: ID does not exist"
Sep 18 22:06:38.779374 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.779347    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"]
Sep 18 22:06:38.785561 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:38.785543    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758526xr"]
Sep 18 22:06:39.618859 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:39.618831    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" path="/var/lib/kubelet/pods/5140ebfe-716f-4ca6-9e29-ac09ee4a34cb/volumes"
Sep 18 22:06:42.001790 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:42.001745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:42.099449 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:42.099416    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:48.490364 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:48.490261    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:49.646325 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:49.646286    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:49.955426 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:49.955340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:51.176855 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:51.176820    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:51.277685 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:51.277651    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:54.336549 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:54.336515    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:54.414215 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:54.414186    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:06:57.776275 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:06:57.776224    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:07:03.705069 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:03.705006    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:07:03.806824 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:03.806798    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:07:06.321497 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:06.321464    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:07:23.606720 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.606684    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"]
Sep 18 22:07:23.607208 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.607187    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main"
Sep 18 22:07:23.607314 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.607211    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main"
Sep 18 22:07:23.607376 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.607332    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5140ebfe-716f-4ca6-9e29-ac09ee4a34cb" containerName="main"
Sep 18 22:07:23.610744 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.610722    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.613979 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.613953    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 18 22:07:23.614086 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.614064    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-hkbh2\""
Sep 18 22:07:23.614154 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.614103    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 18 22:07:23.614229 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.614207    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 18 22:07:23.620254 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.620236    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"]
Sep 18 22:07:23.658939 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.658903    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:07:23.748024 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.748000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7m9\" (UniqueName: \"kubernetes.io/projected/6514fe3e-fc4b-4932-be44-86ae74af25d9-kube-api-access-vb7m9\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.748129 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.748029    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-home\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.748129 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.748061    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.748129 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.748087    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6514fe3e-fc4b-4932-be44-86ae74af25d9-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.748237 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.748140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.748237 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.748195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.848879 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.848849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.848879 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.848884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6514fe3e-fc4b-4932-be44-86ae74af25d9-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.849186 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.848906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.849186 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.848961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.849186 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.849031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7m9\" (UniqueName: \"kubernetes.io/projected/6514fe3e-fc4b-4932-be44-86ae74af25d9-kube-api-access-vb7m9\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.849186 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.849055    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-home\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.849433 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.849371    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.849496 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.849473    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-home\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.849545 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.849497    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.851246 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.851227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6514fe3e-fc4b-4932-be44-86ae74af25d9-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.851695 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.851646    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6514fe3e-fc4b-4932-be44-86ae74af25d9-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.853654 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.853632    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"]
Sep 18 22:07:23.857005 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.856951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:23.858355 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.858336    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7m9\" (UniqueName: \"kubernetes.io/projected/6514fe3e-fc4b-4932-be44-86ae74af25d9-kube-api-access-vb7m9\") pod \"fc-concurrency-test-kserve-b5d44467c-nmtj9\" (UID: \"6514fe3e-fc4b-4932-be44-86ae74af25d9\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.859209 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.859185    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-hcwbd\""
Sep 18 22:07:23.865602 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.865579    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"]
Sep 18 22:07:23.890477 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.890450    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:07:23.923311 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.923254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:23.949823 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.949792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkh9p\" (UniqueName: \"kubernetes.io/projected/cea0a11c-8723-4ee5-b1c7-974bb48f243b-kube-api-access-rkh9p\") pod \"fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk\" (UID: \"cea0a11c-8723-4ee5-b1c7-974bb48f243b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:23.949974 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:23.949846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea0a11c-8723-4ee5-b1c7-974bb48f243b-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk\" (UID: \"cea0a11c-8723-4ee5-b1c7-974bb48f243b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:24.051324 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.051254    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkh9p\" (UniqueName: \"kubernetes.io/projected/cea0a11c-8723-4ee5-b1c7-974bb48f243b-kube-api-access-rkh9p\") pod \"fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk\" (UID: \"cea0a11c-8723-4ee5-b1c7-974bb48f243b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:24.051547 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.051418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea0a11c-8723-4ee5-b1c7-974bb48f243b-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk\" (UID: \"cea0a11c-8723-4ee5-b1c7-974bb48f243b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:24.053923 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.053901    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea0a11c-8723-4ee5-b1c7-974bb48f243b-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk\" (UID: \"cea0a11c-8723-4ee5-b1c7-974bb48f243b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:24.059128 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.059109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkh9p\" (UniqueName: \"kubernetes.io/projected/cea0a11c-8723-4ee5-b1c7-974bb48f243b-kube-api-access-rkh9p\") pod \"fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk\" (UID: \"cea0a11c-8723-4ee5-b1c7-974bb48f243b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:24.178611 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.178528    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:24.199484 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.199458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"]
Sep 18 22:07:24.201879 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:07:24.201854    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6514fe3e_fc4b_4932_be44_86ae74af25d9.slice/crio-a8af7f2369cd2fd8b85bd21929107d3fee685ee5bf6d59b53d4540dc040f9e3d WatchSource:0}: Error finding container a8af7f2369cd2fd8b85bd21929107d3fee685ee5bf6d59b53d4540dc040f9e3d: Status 404 returned error can't find the container with id a8af7f2369cd2fd8b85bd21929107d3fee685ee5bf6d59b53d4540dc040f9e3d
Sep 18 22:07:24.460852 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.460823    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"]
Sep 18 22:07:24.462777 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:07:24.462745    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea0a11c_8723_4ee5_b1c7_974bb48f243b.slice/crio-45d7b7fd04e37d2ffe963a9b59a20c76ee26024f3e3c150595b172d375bd79dd WatchSource:0}: Error finding container 45d7b7fd04e37d2ffe963a9b59a20c76ee26024f3e3c150595b172d375bd79dd: Status 404 returned error can't find the container with id 45d7b7fd04e37d2ffe963a9b59a20c76ee26024f3e3c150595b172d375bd79dd
Sep 18 22:07:24.978620 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.978590    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/0.log"
Sep 18 22:07:24.979096 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.978640    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="748bd9d5f87f094838700fbb3f56a44b465d478822cd33cffd62e2fab08611bf" exitCode=1
Sep 18 22:07:24.979096 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.978740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"748bd9d5f87f094838700fbb3f56a44b465d478822cd33cffd62e2fab08611bf"}
Sep 18 22:07:24.979096 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.978799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerStarted","Data":"45d7b7fd04e37d2ffe963a9b59a20c76ee26024f3e3c150595b172d375bd79dd"}
Sep 18 22:07:24.979096 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.978993    2564 scope.go:117] "RemoveContainer" containerID="748bd9d5f87f094838700fbb3f56a44b465d478822cd33cffd62e2fab08611bf"
Sep 18 22:07:24.980528 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:24.980486    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9" event={"ID":"6514fe3e-fc4b-4932-be44-86ae74af25d9","Type":"ContainerStarted","Data":"a8af7f2369cd2fd8b85bd21929107d3fee685ee5bf6d59b53d4540dc040f9e3d"}
Sep 18 22:07:25.988109 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:25.988065    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9" event={"ID":"6514fe3e-fc4b-4932-be44-86ae74af25d9","Type":"ContainerStarted","Data":"6aa32a0e0493cc0bce634a1fac12e03454b39425a4a0052eae77dbe24349e914"}
Sep 18 22:07:25.989989 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:25.989966    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/1.log"
Sep 18 22:07:25.990438 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:25.990422    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/0.log"
Sep 18 22:07:25.990517 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:25.990455    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="68d6249c19787836e51770b5741e5721408b99e00fe7553202c7b4f559c2ba23" exitCode=1
Sep 18 22:07:25.990579 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:25.990531    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"68d6249c19787836e51770b5741e5721408b99e00fe7553202c7b4f559c2ba23"}
Sep 18 22:07:25.990579 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:25.990571    2564 scope.go:117] "RemoveContainer" containerID="748bd9d5f87f094838700fbb3f56a44b465d478822cd33cffd62e2fab08611bf"
Sep 18 22:07:25.990783 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:25.990760    2564 scope.go:117] "RemoveContainer" containerID="68d6249c19787836e51770b5741e5721408b99e00fe7553202c7b4f559c2ba23"
Sep 18 22:07:25.990980 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:07:25.990961    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:07:26.021958 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:26.021908    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9" podStartSLOduration=1.401403321 podStartE2EDuration="3.021892453s" totalImagesPullingTime="1.620489132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:07:26.018940407 +0000 UTC m=+1024.875596058" watchObservedRunningTime="2026-09-18 22:07:26.021892453 +0000 UTC m=+1024.878548105"
Sep 18 22:07:26.923542 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:26.923515    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:26.996758 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:26.996735    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/1.log"
Sep 18 22:07:26.997102 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:26.997063    2564 scope.go:117] "RemoveContainer" containerID="68d6249c19787836e51770b5741e5721408b99e00fe7553202c7b4f559c2ba23"
Sep 18 22:07:26.997249 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:07:26.997232    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:07:33.924227 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:33.924196    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:33.936510 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:33.936482    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:34.039024 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:34.038998    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-nmtj9"
Sep 18 22:07:34.179463 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:34.179382    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:34.179463 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:34.179431    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:34.179834 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:34.179810    2564 scope.go:117] "RemoveContainer" containerID="68d6249c19787836e51770b5741e5721408b99e00fe7553202c7b4f559c2ba23"
Sep 18 22:07:34.180039 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:07:34.180016    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:07:39.086609 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:39.086572    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:07:40.540421 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:40.540388    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:07:40.615885 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:40.615845    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:07:48.614157 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:48.614130    2564 scope.go:117] "RemoveContainer" containerID="68d6249c19787836e51770b5741e5721408b99e00fe7553202c7b4f559c2ba23"
Sep 18 22:07:49.100053 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:49.100017    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/2.log"
Sep 18 22:07:49.100481 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:49.100463    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/1.log"
Sep 18 22:07:49.100594 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:49.100498    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="a9df6523bca509a8ca7feb1599329ace21a6167f8aaf62a5596ba4819caf91be" exitCode=1
Sep 18 22:07:49.100594 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:49.100526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"a9df6523bca509a8ca7feb1599329ace21a6167f8aaf62a5596ba4819caf91be"}
Sep 18 22:07:49.100594 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:49.100567    2564 scope.go:117] "RemoveContainer" containerID="68d6249c19787836e51770b5741e5721408b99e00fe7553202c7b4f559c2ba23"
Sep 18 22:07:49.100959 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:49.100940    2564 scope.go:117] "RemoveContainer" containerID="a9df6523bca509a8ca7feb1599329ace21a6167f8aaf62a5596ba4819caf91be"
Sep 18 22:07:49.101190 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:07:49.101170    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:07:50.106948 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:50.106919    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/2.log"
Sep 18 22:07:54.179516 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:54.179483    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:54.180008 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:54.179539    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:07:54.180008 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:07:54.179820    2564 scope.go:117] "RemoveContainer" containerID="a9df6523bca509a8ca7feb1599329ace21a6167f8aaf62a5596ba4819caf91be"
Sep 18 22:07:54.180128 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:07:54.180011    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:08:04.613381 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:04.613349    2564 scope.go:117] "RemoveContainer" containerID="a9df6523bca509a8ca7feb1599329ace21a6167f8aaf62a5596ba4819caf91be"
Sep 18 22:08:04.613788 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:08:04.613531    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:08:15.613534 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:15.613454    2564 scope.go:117] "RemoveContainer" containerID="a9df6523bca509a8ca7feb1599329ace21a6167f8aaf62a5596ba4819caf91be"
Sep 18 22:08:16.226859 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:16.226832    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/3.log"
Sep 18 22:08:16.227235 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:16.227222    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/2.log"
Sep 18 22:08:16.227317 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:16.227255    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="efd020ca342dca9104cea92e9fd41e29ba1150df9637431648e716b781202161" exitCode=1
Sep 18 22:08:16.227359 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:16.227338    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"efd020ca342dca9104cea92e9fd41e29ba1150df9637431648e716b781202161"}
Sep 18 22:08:16.227439 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:16.227427    2564 scope.go:117] "RemoveContainer" containerID="a9df6523bca509a8ca7feb1599329ace21a6167f8aaf62a5596ba4819caf91be"
Sep 18 22:08:16.227807 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:16.227792    2564 scope.go:117] "RemoveContainer" containerID="efd020ca342dca9104cea92e9fd41e29ba1150df9637431648e716b781202161"
Sep 18 22:08:16.227998 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:08:16.227975    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:08:17.064469 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:17.064431    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:08:17.232906 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:17.232876    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/3.log"
Sep 18 22:08:24.179407 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:24.179374    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:08:24.179793 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:24.179432    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:08:24.179793 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:24.179762    2564 scope.go:117] "RemoveContainer" containerID="efd020ca342dca9104cea92e9fd41e29ba1150df9637431648e716b781202161"
Sep 18 22:08:24.179958 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:08:24.179929    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:08:27.721087 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:27.721034    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:08:27.816582 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:27.816527    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:08:33.818620 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:33.818579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:08:34.022318 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:34.022261    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:08:34.252762 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:34.252669    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:08:39.613770 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:39.613739    2564 scope.go:117] "RemoveContainer" containerID="efd020ca342dca9104cea92e9fd41e29ba1150df9637431648e716b781202161"
Sep 18 22:08:39.614150 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:08:39.613942    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:08:40.019511 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:40.019416    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:08:40.121024 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:40.120992    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:08:45.767095 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:45.767057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:08:53.613755 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:08:53.613719    2564 scope.go:117] "RemoveContainer" containerID="efd020ca342dca9104cea92e9fd41e29ba1150df9637431648e716b781202161"
Sep 18 22:08:53.614133 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:08:53.613958    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:09:04.613720 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:04.613690    2564 scope.go:117] "RemoveContainer" containerID="efd020ca342dca9104cea92e9fd41e29ba1150df9637431648e716b781202161"
Sep 18 22:09:05.450595 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:05.450567    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/4.log"
Sep 18 22:09:05.450990 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:05.450975    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/3.log"
Sep 18 22:09:05.451044 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:05.451012    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c" exitCode=1
Sep 18 22:09:05.451112 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:05.451093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"}
Sep 18 22:09:05.451162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:05.451151    2564 scope.go:117] "RemoveContainer" containerID="efd020ca342dca9104cea92e9fd41e29ba1150df9637431648e716b781202161"
Sep 18 22:09:05.451455 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:05.451434    2564 scope.go:117] "RemoveContainer" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"
Sep 18 22:09:05.451662 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:09:05.451643    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:09:06.456603 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:06.456574    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/4.log"
Sep 18 22:09:14.179231 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:14.179199    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:09:14.179638 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:14.179256    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:09:14.179638 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:14.179595    2564 scope.go:117] "RemoveContainer" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"
Sep 18 22:09:14.179790 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:09:14.179771    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:09:28.613840 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:28.613808    2564 scope.go:117] "RemoveContainer" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"
Sep 18 22:09:28.614208 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:09:28.613948    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:09:41.615415 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:41.615386    2564 scope.go:117] "RemoveContainer" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"
Sep 18 22:09:41.616040 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:09:41.615562    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:09:54.613975 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:09:54.613885    2564 scope.go:117] "RemoveContainer" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"
Sep 18 22:09:54.614455 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:09:54.614083    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:10:07.614025 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:07.613996    2564 scope.go:117] "RemoveContainer" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"
Sep 18 22:10:07.614437 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:10:07.614192    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:10:21.650778 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:21.650744    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/4.log"
Sep 18 22:10:21.654515 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:21.654496    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/4.log"
Sep 18 22:10:21.656045 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:21.656023    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:10:21.659200 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:21.659184    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:10:22.613660 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:22.613632    2564 scope.go:117] "RemoveContainer" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"
Sep 18 22:10:22.613825 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:10:22.613801    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:10:34.613228 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:34.613199    2564 scope.go:117] "RemoveContainer" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"
Sep 18 22:10:34.863779 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:34.863707    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/5.log"
Sep 18 22:10:34.864099 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:34.864084    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/4.log"
Sep 18 22:10:34.864181 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:34.864115    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2" exitCode=1
Sep 18 22:10:34.864235 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:34.864197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"}
Sep 18 22:10:34.864311 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:34.864297    2564 scope.go:117] "RemoveContainer" containerID="3a7fbcc218e3a64124278fdb206d7a99e28d544539d49c2dbbfbd49af59a649c"
Sep 18 22:10:34.864575 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:34.864558    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:10:34.864795 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:10:34.864772    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:10:35.869963 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:35.869934    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/5.log"
Sep 18 22:10:44.178980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:44.178930    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:10:44.179418 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:44.179004    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:10:44.179418 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:44.179342    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:10:44.179556 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:10:44.179537    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:10:55.614226 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:10:55.614194    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:10:55.614644 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:10:55.614406    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:11:07.613578 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:11:07.613552    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:11:07.614030 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:11:07.613733    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:11:10.768609 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:11:10.768578    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:11:19.619545 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:11:19.619476    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:11:19.619894 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:11:19.619666    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:11:34.613613 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:11:34.613583    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:11:34.614009 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:11:34.613743    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:11:36.611434 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:11:36.611401    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:11:36.709077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:11:36.709035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:11:45.613916 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:11:45.613877    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:11:45.614554 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:11:45.614083    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:11:59.613816 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:11:59.613780    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:11:59.614338 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:11:59.614097    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:12:10.614021 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:12:10.613977    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:12:10.614535 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:12:10.614234    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:12:15.375563 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:12:15.375531    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:12:15.445153 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:12:15.445120    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:12:25.613690 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:12:25.613655    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:12:25.614116 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:12:25.613876    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:12:39.614098 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:12:39.614063    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:12:39.614518 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:12:39.614305    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:12:51.616101 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:12:51.616024    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:12:51.616507 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:12:51.616180    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:13:04.614221 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:04.614195    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:13:04.614850 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:13:04.614431    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:13:17.614222 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:17.614182    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:13:18.635661 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:18.635634    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/6.log"
Sep 18 22:13:18.636095 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:18.636078    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/5.log"
Sep 18 22:13:18.636140 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:18.636112    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55" exitCode=1
Sep 18 22:13:18.636203 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:18.636185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"}
Sep 18 22:13:18.636303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:18.636289    2564 scope.go:117] "RemoveContainer" containerID="7177c39569a56af5f0390e96ce5f8c4c051debddb4d68ac708424f2f6c8a2ba2"
Sep 18 22:13:18.636574 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:18.636552    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:13:18.636757 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:13:18.636739    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:13:19.641680 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:19.641653    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/6.log"
Sep 18 22:13:24.179307 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:24.179236    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:13:24.179724 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:24.179331    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:13:24.179724 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:24.179668    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:13:24.179886 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:13:24.179865    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:13:35.614035 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:35.614002    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:13:35.614508 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:13:35.614211    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:13:46.614193 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:13:46.614164    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:13:46.614695 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:13:46.614374    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:14:00.613765 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:14:00.613737    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:14:00.614147 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:14:00.613905    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:14:05.938362 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:14:05.938323    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:14:15.614289 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:14:15.614204    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:14:15.614656 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:14:15.614414    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:14:26.613644 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:14:26.613609    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:14:26.614090 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:14:26.613789    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:14:41.615954 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:14:41.615926    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:14:41.616337 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:14:41.616110    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:14:52.613649 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:14:52.613620    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:14:52.614011 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:14:52.613805    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:15:07.613613 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:15:07.613578    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:15:07.614081 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:15:07.613763    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:15:19.614181 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:15:19.614148    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:15:19.614560 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:15:19.614404    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:15:21.679620 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:15:21.679528    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/6.log"
Sep 18 22:15:21.684301 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:15:21.684284    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:15:21.690738 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:15:21.690720    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/6.log"
Sep 18 22:15:21.695338 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:15:21.695321    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:15:34.614479 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:15:34.614446    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:15:34.614833 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:15:34.614594    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:15:48.614301 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:15:48.614208    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:15:48.614749 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:15:48.614444    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:15:59.614164 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:15:59.614136    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:15:59.614544 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:15:59.614365    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:16:13.614017 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:16:13.613984    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:16:13.614514 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:16:13.614185    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:16:27.613765 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:16:27.613737    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:16:27.614220 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:16:27.613907    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:16:39.613572 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:16:39.613534    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:16:39.614081 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:16:39.613688    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:16:50.613773 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:16:50.613745    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:16:50.614221 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:16:50.613922    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:17:04.613615 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:17:04.613587    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:17:04.614070 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:17:04.613758    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:17:16.613435 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:17:16.613365    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:17:16.613790 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:17:16.613594    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:17:31.615701 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:17:31.615674    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:17:31.616180 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:17:31.615867    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:17:46.613463 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:17:46.613436    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:17:46.613911 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:17:46.613628    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:17:58.613623 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:17:58.613589    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:17:58.613974 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:17:58.613838    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:18:11.615950 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:11.615922    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:18:11.616426 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:18:11.616152    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:18:24.613395 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:24.613358    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:18:25.009767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:25.009699    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/7.log"
Sep 18 22:18:25.010122 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:25.010103    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/6.log"
Sep 18 22:18:25.010238 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:25.010143    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e" exitCode=1
Sep 18 22:18:25.010238 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:25.010173    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"}
Sep 18 22:18:25.010238 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:25.010231    2564 scope.go:117] "RemoveContainer" containerID="af8755f1f12e79dd0c25e4326098669c8d9031fc882025a22c6ccd2b444ecf55"
Sep 18 22:18:25.010576 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:25.010553    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:18:25.010808 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:18:25.010784    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:18:26.016219 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:26.016197    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/7.log"
Sep 18 22:18:34.178807 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:34.178773    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:18:34.179341 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:34.178829    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:18:34.179341 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:34.179125    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:18:34.179341 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:18:34.179325    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:18:47.614066 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:47.613993    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:18:47.614438 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:18:47.614196    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:18:59.614061 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:18:59.614030    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:18:59.614461 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:18:59.614202    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:19:11.616260 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:19:11.616231    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:19:11.616808 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:19:11.616448    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:19:26.614211 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:19:26.614179    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:19:26.614611 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:19:26.614379    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:19:37.619230 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:19:37.619191    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:19:37.619680 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:19:37.619437    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:19:48.455063 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:19:48.455029    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:19:50.614206 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:19:50.614172    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:19:50.614674 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:19:50.614452    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:20:04.613957 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:20:04.613927    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:20:04.614341 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:20:04.614116    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:20:15.613955 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:20:15.613870    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:20:15.614392 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:20:15.614117    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:20:21.707940 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:20:21.707835    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/7.log"
Sep 18 22:20:21.713417 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:20:21.713397    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:20:21.720309 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:20:21.720287    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/7.log"
Sep 18 22:20:21.725007 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:20:21.724985    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:20:27.614291 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:20:27.614225    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:20:27.614694 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:20:27.614456    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:20:41.616244 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:20:41.616211    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:20:41.616665 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:20:41.616417    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:20:52.614080 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:20:52.614051    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:20:52.614589 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:20:52.614213    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:21:03.613710 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:21:03.613678    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:21:03.614188 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:21:03.613840    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:21:17.613584 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:21:17.613544    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:21:17.614065 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:21:17.613782    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:21:32.614001 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:21:32.613969    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:21:32.614536 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:21:32.614167    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:21:45.614316 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:21:45.614196    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:21:45.614777 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:21:45.614489    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:21:53.274488 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:21:53.274436    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:21:53.360383 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:21:53.360353    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:21:53.451437 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:21:53.451412    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:21:53.528238 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:21:53.528208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:21:56.809303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:21:56.809249    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:00.613626 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:00.613597    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:22:00.614010 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:22:00.613779    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:22:13.614290 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:13.614240    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:22:13.614776 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:22:13.614480    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:22:18.176240 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:18.176203    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:21.036014 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.035984    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:21.134752 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.134720    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:21.235991 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.235961    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:21.332199 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.332124    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"]
Sep 18 22:22:21.335655 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.335630    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.344515 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.344489    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 18 22:22:21.344884 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.344864    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 18 22:22:21.345116 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.345098    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 18 22:22:21.345639 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.345620    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-hhlmv\""
Sep 18 22:22:21.363253 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.363229    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"]
Sep 18 22:22:21.414775 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.414753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:21.450080 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.450058    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.450169 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.450086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c340343-1234-4aa1-a80a-3a10973b3dd4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.450169 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.450117    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.450299 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.450177    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.450299 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.450200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.450299 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.450220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m54ql\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-kube-api-access-m54ql\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.450299 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.450241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.450299 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.450279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.450559 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.450364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.530646 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.530619    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:21.551573 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.551549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c340343-1234-4aa1-a80a-3a10973b3dd4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.551671 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.551590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.551671 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.551625    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.551671 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.551642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.551671 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.551666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m54ql\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-kube-api-access-m54ql\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.551890 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.551873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.551940 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.551913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.552003 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.551945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.552069 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.552014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.552069 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.552041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.552172 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.552123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.552252 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.552234    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.552348 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.552334    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.554082 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.554058    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.554182 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.554154    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.554773 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.554759    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 18 22:22:21.562600 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.562579    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 18 22:22:21.562600 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.562591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c340343-1234-4aa1-a80a-3a10973b3dd4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.564471 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.564453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.573058 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.573039    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 18 22:22:21.582932 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.582885    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m54ql\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-kube-api-access-m54ql\") pod \"router-gateway-1-openshift-default-7dcbd489bd-96gkv\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.649718 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.649693    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-hhlmv\""
Sep 18 22:22:21.657770 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.657754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:21.931986 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.931964    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"]
Sep 18 22:22:21.933812 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:22:21.933789    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c340343_1234_4aa1_a80a_3a10973b3dd4.slice/crio-eaf05d5f7742af66cd3bab0662fedb82fb63c2fe9c81a0b24c75969474044b8b WatchSource:0}: Error finding container eaf05d5f7742af66cd3bab0662fedb82fb63c2fe9c81a0b24c75969474044b8b: Status 404 returned error can't find the container with id eaf05d5f7742af66cd3bab0662fedb82fb63c2fe9c81a0b24c75969474044b8b
Sep 18 22:22:21.935775 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.935743    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 22:22:21.935868 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.935809    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 22:22:21.935868 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:21.935844    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 22:22:22.113943 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:22.113903    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv" event={"ID":"5c340343-1234-4aa1-a80a-3a10973b3dd4","Type":"ContainerStarted","Data":"c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09"}
Sep 18 22:22:22.114350 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:22.113971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv" event={"ID":"5c340343-1234-4aa1-a80a-3a10973b3dd4","Type":"ContainerStarted","Data":"eaf05d5f7742af66cd3bab0662fedb82fb63c2fe9c81a0b24c75969474044b8b"}
Sep 18 22:22:22.138102 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:22.138049    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv" podStartSLOduration=1.138029196 podStartE2EDuration="1.138029196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:22:22.136608752 +0000 UTC m=+1920.993264404" watchObservedRunningTime="2026-09-18 22:22:22.138029196 +0000 UTC m=+1920.994684848"
Sep 18 22:22:22.658139 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:22.658103    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:22.663087 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:22.663063    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:23.119100 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:23.119061    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:23.120077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:23.120053    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:24.314448 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:24.314413    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:26.294434 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:26.294400    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:27.229529 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:27.229499    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:28.614184 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:28.614150    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:22:28.614604 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:22:28.614434    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:22:31.439014 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:31.438979    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:33.012467 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:33.012430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:37.866531 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:37.866495    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:39.169843 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:39.169806    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:39.265285 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:39.265230    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:40.108388 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:40.108350    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:43.534390 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:43.534358    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:43.614063 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:43.614036    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:22:43.614262 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:22:43.614241    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:22:43.636120 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:43.636088    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:45.396160 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:45.396119    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:48.514096 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:48.514050    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"]
Sep 18 22:22:48.514609 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:48.514399    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv" podUID="5c340343-1234-4aa1-a80a-3a10973b3dd4" containerName="istio-proxy" containerID="cri-o://c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09" gracePeriod=30
Sep 18 22:22:48.612727 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:48.612691    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:50.157079 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.157054    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:50.236978 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.236944    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-podinfo\") pod \"5c340343-1234-4aa1-a80a-3a10973b3dd4\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") "
Sep 18 22:22:50.237162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237033    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-credential-socket\") pod \"5c340343-1234-4aa1-a80a-3a10973b3dd4\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") "
Sep 18 22:22:50.237162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237067    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-data\") pod \"5c340343-1234-4aa1-a80a-3a10973b3dd4\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") "
Sep 18 22:22:50.237162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237102    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-certs\") pod \"5c340343-1234-4aa1-a80a-3a10973b3dd4\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") "
Sep 18 22:22:50.237162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237150    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-token\") pod \"5c340343-1234-4aa1-a80a-3a10973b3dd4\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") "
Sep 18 22:22:50.237402 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237186    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m54ql\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-kube-api-access-m54ql\") pod \"5c340343-1234-4aa1-a80a-3a10973b3dd4\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") "
Sep 18 22:22:50.237402 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237243    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-socket\") pod \"5c340343-1234-4aa1-a80a-3a10973b3dd4\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") "
Sep 18 22:22:50.237402 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237301    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c340343-1234-4aa1-a80a-3a10973b3dd4-istiod-ca-cert\") pod \"5c340343-1234-4aa1-a80a-3a10973b3dd4\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") "
Sep 18 22:22:50.237402 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237327    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-envoy\") pod \"5c340343-1234-4aa1-a80a-3a10973b3dd4\" (UID: \"5c340343-1234-4aa1-a80a-3a10973b3dd4\") "
Sep 18 22:22:50.237592 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237427    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c340343-1234-4aa1-a80a-3a10973b3dd4" (UID: "5c340343-1234-4aa1-a80a-3a10973b3dd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:50.237592 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237467    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c340343-1234-4aa1-a80a-3a10973b3dd4" (UID: "5c340343-1234-4aa1-a80a-3a10973b3dd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:50.237708 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237687    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c340343-1234-4aa1-a80a-3a10973b3dd4" (UID: "5c340343-1234-4aa1-a80a-3a10973b3dd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:50.237708 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237696    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c340343-1234-4aa1-a80a-3a10973b3dd4" (UID: "5c340343-1234-4aa1-a80a-3a10973b3dd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:50.237806 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237792    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-socket\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:22:50.237857 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237814    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-credential-socket\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:22:50.237857 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237831    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-data\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:22:50.237857 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237844    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-workload-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:22:50.238012 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.237982    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c340343-1234-4aa1-a80a-3a10973b3dd4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c340343-1234-4aa1-a80a-3a10973b3dd4" (UID: "5c340343-1234-4aa1-a80a-3a10973b3dd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:22:50.239979 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.239940    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c340343-1234-4aa1-a80a-3a10973b3dd4" (UID: "5c340343-1234-4aa1-a80a-3a10973b3dd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:22:50.239979 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.239948    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c340343-1234-4aa1-a80a-3a10973b3dd4" (UID: "5c340343-1234-4aa1-a80a-3a10973b3dd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:22:50.240137 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.240038    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c340343-1234-4aa1-a80a-3a10973b3dd4" (UID: "5c340343-1234-4aa1-a80a-3a10973b3dd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:50.240137 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.240117    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-kube-api-access-m54ql" (OuterVolumeSpecName: "kube-api-access-m54ql") pod "5c340343-1234-4aa1-a80a-3a10973b3dd4" (UID: "5c340343-1234-4aa1-a80a-3a10973b3dd4"). InnerVolumeSpecName "kube-api-access-m54ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:22:50.256141 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.256109    2564 generic.go:356] "Generic (PLEG): container finished" podID="5c340343-1234-4aa1-a80a-3a10973b3dd4" containerID="c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09" exitCode=0
Sep 18 22:22:50.256255 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.256158    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"
Sep 18 22:22:50.256255 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.256186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv" event={"ID":"5c340343-1234-4aa1-a80a-3a10973b3dd4","Type":"ContainerDied","Data":"c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09"}
Sep 18 22:22:50.256390 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.256257    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv" event={"ID":"5c340343-1234-4aa1-a80a-3a10973b3dd4","Type":"ContainerDied","Data":"eaf05d5f7742af66cd3bab0662fedb82fb63c2fe9c81a0b24c75969474044b8b"}
Sep 18 22:22:50.256390 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.256301    2564 scope.go:117] "RemoveContainer" containerID="c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09"
Sep 18 22:22:50.265912 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.265893    2564 scope.go:117] "RemoveContainer" containerID="c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09"
Sep 18 22:22:50.266179 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:22:50.266160    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09\": container with ID starting with c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09 not found: ID does not exist" containerID="c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09"
Sep 18 22:22:50.266242 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.266192    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09"} err="failed to get container status \"c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09\": rpc error: code = NotFound desc = could not find container \"c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09\": container with ID starting with c3c5c1802825f118c815902d5b4d3ddb7091d07d7748aaa164e38102ba350d09 not found: ID does not exist"
Sep 18 22:22:50.298732 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.298692    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"]
Sep 18 22:22:50.307441 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.307412    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-96gkv"]
Sep 18 22:22:50.339251 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.339220    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-token\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:22:50.339251 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.339245    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m54ql\" (UniqueName: \"kubernetes.io/projected/5c340343-1234-4aa1-a80a-3a10973b3dd4-kube-api-access-m54ql\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:22:50.339251 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.339257    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c340343-1234-4aa1-a80a-3a10973b3dd4-istiod-ca-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:22:50.339489 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.339288    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-envoy\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:22:50.339489 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:50.339296    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c340343-1234-4aa1-a80a-3a10973b3dd4-istio-podinfo\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:22:51.619673 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:51.619638    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c340343-1234-4aa1-a80a-3a10973b3dd4" path="/var/lib/kubelet/pods/5c340343-1234-4aa1-a80a-3a10973b3dd4/volumes"
Sep 18 22:22:54.870324 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:54.870292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:55.471178 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:55.471146    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:22:58.613578 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:22:58.613552    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:22:58.613944 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:22:58.613710    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:23:01.133539 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:01.133501    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:09.613250 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:09.613220    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:23:09.613757 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:23:09.613430    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:23:10.223539 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:10.223505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:10.292567 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:10.292537    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:23.385761 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:23.385726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:23.613675 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:23.613646    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:23:23.613838 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:23:23.613823    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:23:23.784517 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:23.784488    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:30.759756 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:30.759714    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:30.858794 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:30.858754    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:34.613991 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:34.613954    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:23:35.474343 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:35.474314    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/8.log"
Sep 18 22:23:35.474767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:35.474752    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/7.log"
Sep 18 22:23:35.474821 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:35.474789    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174" exitCode=1
Sep 18 22:23:35.474898 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:35.474875    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"}
Sep 18 22:23:35.474994 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:35.474953    2564 scope.go:117] "RemoveContainer" containerID="b140d0036852779381e40b76e48cd22710196fe478f3dccbfed4df935defa92e"
Sep 18 22:23:35.475353 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:35.475334    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:23:35.475582 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:23:35.475561    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:23:36.481100 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:36.481066    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/8.log"
Sep 18 22:23:36.869419 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:36.869385    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:44.179604 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:44.179544    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:23:44.180180 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:44.179630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:23:44.180180 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:44.179953    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:23:44.180327 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:23:44.180178    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:23:51.071429 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.071391    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"]
Sep 18 22:23:51.071967 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.071940    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c340343-1234-4aa1-a80a-3a10973b3dd4" containerName="istio-proxy"
Sep 18 22:23:51.071967 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.071966    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c340343-1234-4aa1-a80a-3a10973b3dd4" containerName="istio-proxy"
Sep 18 22:23:51.072128 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.072042    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c340343-1234-4aa1-a80a-3a10973b3dd4" containerName="istio-proxy"
Sep 18 22:23:51.076244 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.076225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.078908 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.078885    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 22:23:51.079034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.078997    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 22:23:51.079818 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.079799    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 22:23:51.079818 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.079813    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-nwkkw\""
Sep 18 22:23:51.086591 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.086571    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"]
Sep 18 22:23:51.134636 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.134607    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:51.168534 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.168492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89dd3a0-382a-4e1a-b6fb-d012198819ef-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.168709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.168548    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.168709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.168651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.168709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.168685    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.168709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.168705    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9bm8\" (UniqueName: \"kubernetes.io/projected/b89dd3a0-382a-4e1a-b6fb-d012198819ef-kube-api-access-s9bm8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.168853 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.168816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.269619 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.269577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.269619 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.269617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.269885 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.269635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9bm8\" (UniqueName: \"kubernetes.io/projected/b89dd3a0-382a-4e1a-b6fb-d012198819ef-kube-api-access-s9bm8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.269885 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.269699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.269885 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.269718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89dd3a0-382a-4e1a-b6fb-d012198819ef-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.269885 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.269751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.270124 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.270098    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.270181 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.270129    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.270181 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.270158    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.272097 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.272078    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89dd3a0-382a-4e1a-b6fb-d012198819ef-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.272180 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.272164    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89dd3a0-382a-4e1a-b6fb-d012198819ef-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.280525 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.280505    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9bm8\" (UniqueName: \"kubernetes.io/projected/b89dd3a0-382a-4e1a-b6fb-d012198819ef-kube-api-access-s9bm8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2\" (UID: \"b89dd3a0-382a-4e1a-b6fb-d012198819ef\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.387619 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.387518    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:51.411522 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.411474    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"]
Sep 18 22:23:51.415392 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.415365    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:23:51.418289 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.418248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-2wdp9\""
Sep 18 22:23:51.429555 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.429524    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"]
Sep 18 22:23:51.442413 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.442387    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:23:51.471840 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.471802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b30dbc3-fb8a-439e-98c4-73c6cd5f7864-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr\" (UID: \"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:23:51.472004 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.471887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjng\" (UniqueName: \"kubernetes.io/projected/9b30dbc3-fb8a-439e-98c4-73c6cd5f7864-kube-api-access-ctjng\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr\" (UID: \"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:23:51.573368 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.573334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b30dbc3-fb8a-439e-98c4-73c6cd5f7864-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr\" (UID: \"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:23:51.573540 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.573400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjng\" (UniqueName: \"kubernetes.io/projected/9b30dbc3-fb8a-439e-98c4-73c6cd5f7864-kube-api-access-ctjng\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr\" (UID: \"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:23:51.575896 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.575867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b30dbc3-fb8a-439e-98c4-73c6cd5f7864-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr\" (UID: \"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:23:51.584047 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.584022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjng\" (UniqueName: \"kubernetes.io/projected/9b30dbc3-fb8a-439e-98c4-73c6cd5f7864-kube-api-access-ctjng\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr\" (UID: \"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:23:51.669516 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.669439    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"]
Sep 18 22:23:51.672511 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:23:51.672469    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89dd3a0_382a_4e1a_b6fb_d012198819ef.slice/crio-f698ce33ff257d4dc0436e491c42886be6818d0ac29d83877760b44e164c1cee WatchSource:0}: Error finding container f698ce33ff257d4dc0436e491c42886be6818d0ac29d83877760b44e164c1cee: Status 404 returned error can't find the container with id f698ce33ff257d4dc0436e491c42886be6818d0ac29d83877760b44e164c1cee
Sep 18 22:23:51.727006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:51.726982    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:23:52.001621 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:52.001593    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"]
Sep 18 22:23:52.003854 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:23:52.003810    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b30dbc3_fb8a_439e_98c4_73c6cd5f7864.slice/crio-243335f80be12ce07ca52f092e5aa9d4c8c9a13f2c1a777744cd1ae27085f095 WatchSource:0}: Error finding container 243335f80be12ce07ca52f092e5aa9d4c8c9a13f2c1a777744cd1ae27085f095: Status 404 returned error can't find the container with id 243335f80be12ce07ca52f092e5aa9d4c8c9a13f2c1a777744cd1ae27085f095
Sep 18 22:23:52.551523 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:52.551490    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/0.log"
Sep 18 22:23:52.551982 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:52.551535    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="c2c06f343162c93f746630c273aa4edfc1cd215276f0c6b50f5ecfdc3e596997" exitCode=1
Sep 18 22:23:52.551982 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:52.551637    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"c2c06f343162c93f746630c273aa4edfc1cd215276f0c6b50f5ecfdc3e596997"}
Sep 18 22:23:52.551982 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:52.551706    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerStarted","Data":"243335f80be12ce07ca52f092e5aa9d4c8c9a13f2c1a777744cd1ae27085f095"}
Sep 18 22:23:52.551982 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:52.551870    2564 scope.go:117] "RemoveContainer" containerID="c2c06f343162c93f746630c273aa4edfc1cd215276f0c6b50f5ecfdc3e596997"
Sep 18 22:23:52.553552 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:52.553530    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2" event={"ID":"b89dd3a0-382a-4e1a-b6fb-d012198819ef","Type":"ContainerStarted","Data":"063c955209a3a358dfd8f89e9da83764b997254f8a64049c97079cafd7fcfdf4"}
Sep 18 22:23:52.553649 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:52.553572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2" event={"ID":"b89dd3a0-382a-4e1a-b6fb-d012198819ef","Type":"ContainerStarted","Data":"f698ce33ff257d4dc0436e491c42886be6818d0ac29d83877760b44e164c1cee"}
Sep 18 22:23:52.608761 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:52.608709    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2" podStartSLOduration=1.608694536 podStartE2EDuration="1.608694536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:23:52.607511162 +0000 UTC m=+2011.464166826" watchObservedRunningTime="2026-09-18 22:23:52.608694536 +0000 UTC m=+2011.465350188"
Sep 18 22:23:53.388293 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:53.388234    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:23:53.560354 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:53.560324    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/1.log"
Sep 18 22:23:53.560766 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:53.560748    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/0.log"
Sep 18 22:23:53.560811 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:53.560783    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="ccc979c037b91474907eb673b30ae85190b8e0033af4d06a824fd49d7d64b54c" exitCode=1
Sep 18 22:23:53.560907 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:53.560880    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"ccc979c037b91474907eb673b30ae85190b8e0033af4d06a824fd49d7d64b54c"}
Sep 18 22:23:53.560959 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:53.560945    2564 scope.go:117] "RemoveContainer" containerID="c2c06f343162c93f746630c273aa4edfc1cd215276f0c6b50f5ecfdc3e596997"
Sep 18 22:23:53.561111 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:53.561084    2564 scope.go:117] "RemoveContainer" containerID="ccc979c037b91474907eb673b30ae85190b8e0033af4d06a824fd49d7d64b54c"
Sep 18 22:23:53.561372 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:23:53.561351    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:23:54.567442 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:54.567415    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/1.log"
Sep 18 22:23:54.567809 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:54.567746    2564 scope.go:117] "RemoveContainer" containerID="ccc979c037b91474907eb673b30ae85190b8e0033af4d06a824fd49d7d64b54c"
Sep 18 22:23:54.567967 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:23:54.567947    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:23:56.613275 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:23:56.613237    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:23:56.613652 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:23:56.613480    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:24:01.388217 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:01.388178    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:24:01.400654 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:01.400626    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:24:01.619192 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:01.619166    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2"
Sep 18 22:24:01.727157 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:01.727062    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:24:01.727157 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:01.727138    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:24:01.727613 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:01.727594    2564 scope.go:117] "RemoveContainer" containerID="ccc979c037b91474907eb673b30ae85190b8e0033af4d06a824fd49d7d64b54c"
Sep 18 22:24:01.727853 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:01.727821    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:24:07.613939 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:07.613909    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:24:07.614383 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:07.614146    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:24:16.613753 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:16.613721    2564 scope.go:117] "RemoveContainer" containerID="ccc979c037b91474907eb673b30ae85190b8e0033af4d06a824fd49d7d64b54c"
Sep 18 22:24:17.679085 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:17.679052    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/2.log"
Sep 18 22:24:17.679550 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:17.679534    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/1.log"
Sep 18 22:24:17.679596 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:17.679572    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="ba7061c515f2881fd0e623a70d76d390d85d53ecb7c6a76580107cbbee4724f0" exitCode=1
Sep 18 22:24:17.679671 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:17.679645    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"ba7061c515f2881fd0e623a70d76d390d85d53ecb7c6a76580107cbbee4724f0"}
Sep 18 22:24:17.679721 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:17.679710    2564 scope.go:117] "RemoveContainer" containerID="ccc979c037b91474907eb673b30ae85190b8e0033af4d06a824fd49d7d64b54c"
Sep 18 22:24:17.680043 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:17.680025    2564 scope.go:117] "RemoveContainer" containerID="ba7061c515f2881fd0e623a70d76d390d85d53ecb7c6a76580107cbbee4724f0"
Sep 18 22:24:17.680238 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:17.680220    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:24:18.224955 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:18.224918    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:24:18.686606 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:18.686577    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/2.log"
Sep 18 22:24:20.613576 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:20.613544    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:24:20.613961 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:20.613733    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:24:21.727319 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:21.727284    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:24:21.727658 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:21.727353    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:24:21.727716 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:21.727675    2564 scope.go:117] "RemoveContainer" containerID="ba7061c515f2881fd0e623a70d76d390d85d53ecb7c6a76580107cbbee4724f0"
Sep 18 22:24:21.727888 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:21.727868    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:24:33.613239 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:33.613210    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:24:33.613633 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:33.613381    2564 scope.go:117] "RemoveContainer" containerID="ba7061c515f2881fd0e623a70d76d390d85d53ecb7c6a76580107cbbee4724f0"
Sep 18 22:24:33.613633 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:33.613422    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:24:33.613633 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:33.613535    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:24:44.614066 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:44.613989    2564 scope.go:117] "RemoveContainer" containerID="ba7061c515f2881fd0e623a70d76d390d85d53ecb7c6a76580107cbbee4724f0"
Sep 18 22:24:44.811951 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:44.811918    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/3.log"
Sep 18 22:24:44.812369 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:44.812352    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/2.log"
Sep 18 22:24:44.812449 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:44.812390    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="668598f795851709fb92413af7d68a192e37e435a6b932b3d4dff8415aa2c66c" exitCode=1
Sep 18 22:24:44.812488 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:44.812472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"668598f795851709fb92413af7d68a192e37e435a6b932b3d4dff8415aa2c66c"}
Sep 18 22:24:44.812544 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:44.812531    2564 scope.go:117] "RemoveContainer" containerID="ba7061c515f2881fd0e623a70d76d390d85d53ecb7c6a76580107cbbee4724f0"
Sep 18 22:24:44.812831 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:44.812810    2564 scope.go:117] "RemoveContainer" containerID="668598f795851709fb92413af7d68a192e37e435a6b932b3d4dff8415aa2c66c"
Sep 18 22:24:44.813041 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:44.813021    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:24:45.613725 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:45.613692    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:24:45.613933 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:45.613893    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:24:45.824787 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:45.824759    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/3.log"
Sep 18 22:24:51.727709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:51.727679    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:24:51.728184 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:51.727739    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:24:51.728184 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:51.728092    2564 scope.go:117] "RemoveContainer" containerID="668598f795851709fb92413af7d68a192e37e435a6b932b3d4dff8415aa2c66c"
Sep 18 22:24:51.728350 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:51.728327    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:24:57.619117 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:24:57.619090    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:24:57.619523 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:24:57.619298    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:25:03.614018 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:03.613982    2564 scope.go:117] "RemoveContainer" containerID="668598f795851709fb92413af7d68a192e37e435a6b932b3d4dff8415aa2c66c"
Sep 18 22:25:03.614391 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:03.614219    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:25:10.613291 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:10.613240    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:25:10.613745 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:10.613496    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:25:18.613618 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:18.613591    2564 scope.go:117] "RemoveContainer" containerID="668598f795851709fb92413af7d68a192e37e435a6b932b3d4dff8415aa2c66c"
Sep 18 22:25:18.614016 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:18.613781    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:25:21.738600 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:21.738568    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/3.log"
Sep 18 22:25:21.739363 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:21.739348    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/8.log"
Sep 18 22:25:21.744104 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:21.744085    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:25:21.750378 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:21.750361    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/3.log"
Sep 18 22:25:21.751109 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:21.751089    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/8.log"
Sep 18 22:25:21.756003 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:21.755978    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:25:22.613979 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:22.613952    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:25:22.614207 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:22.614132    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:25:33.613570 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:33.613540    2564 scope.go:117] "RemoveContainer" containerID="668598f795851709fb92413af7d68a192e37e435a6b932b3d4dff8415aa2c66c"
Sep 18 22:25:33.613948 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:33.613611    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:25:33.613948 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:33.613759    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:25:34.057801 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:34.057774    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/4.log"
Sep 18 22:25:34.058215 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:34.058198    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/3.log"
Sep 18 22:25:34.058287 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:34.058234    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947" exitCode=1
Sep 18 22:25:34.058334 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:34.058314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"}
Sep 18 22:25:34.058382 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:34.058369    2564 scope.go:117] "RemoveContainer" containerID="668598f795851709fb92413af7d68a192e37e435a6b932b3d4dff8415aa2c66c"
Sep 18 22:25:34.058656 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:34.058644    2564 scope.go:117] "RemoveContainer" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"
Sep 18 22:25:34.058873 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:34.058852    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:25:35.065219 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:35.065190    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/4.log"
Sep 18 22:25:41.727586 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:41.727552    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:25:41.728070 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:41.727612    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:25:41.728070 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:41.727958    2564 scope.go:117] "RemoveContainer" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"
Sep 18 22:25:41.728201 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:41.728178    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:25:45.614137 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:45.614108    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:25:45.614517 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:45.614294    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:25:56.613294 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:56.613232    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:25:56.613853 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:56.613466    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:25:57.613804 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:57.613775    2564 scope.go:117] "RemoveContainer" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"
Sep 18 22:25:57.614302 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:25:57.613954    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:25:59.140154 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:25:59.140110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:26:00.514077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:00.514041    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:26:08.614285 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:08.614247    2564 scope.go:117] "RemoveContainer" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"
Sep 18 22:26:08.614639 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:26:08.614499    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:26:09.619618 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:09.619587    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:26:09.620064 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:26:09.619815    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:26:22.613432 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:22.613360    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:26:22.613779 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:26:22.613542    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:26:23.618985 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:23.618961    2564 scope.go:117] "RemoveContainer" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"
Sep 18 22:26:23.619356 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:26:23.619127    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:26:35.618923 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:35.618892    2564 scope.go:117] "RemoveContainer" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"
Sep 18 22:26:35.619403 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:35.619026    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:26:35.619403 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:26:35.619094    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:26:35.619403 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:26:35.619181    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:26:49.108549 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:49.108520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:26:49.211100 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:49.211074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:26:49.613667 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:49.613640    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:26:49.613860 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:49.613733    2564 scope.go:117] "RemoveContainer" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"
Sep 18 22:26:49.613930 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:26:49.613871    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:26:49.613930 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:26:49.613874    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:26:51.671661 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:51.671627    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:26:51.933158 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:51.933083    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:26:56.187358 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:26:56.187326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:27:00.996354 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:00.996321    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:27:02.262486 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:02.262448    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:27:02.363179 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:02.363143    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:27:02.613864 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:02.613835    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:27:02.614031 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:02.614012    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:27:04.613244 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:04.613216    2564 scope.go:117] "RemoveContainer" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"
Sep 18 22:27:05.490714 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:05.490688    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/5.log"
Sep 18 22:27:05.491117 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:05.491102    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/4.log"
Sep 18 22:27:05.491179 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:05.491138    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5" exitCode=1
Sep 18 22:27:05.491238 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:05.491218    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"}
Sep 18 22:27:05.491308 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:05.491297    2564 scope.go:117] "RemoveContainer" containerID="cc05d1c2462b8269a5a379bc1c7b5b5ad62bcf017c561ea3d12095416610c947"
Sep 18 22:27:05.491682 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:05.491666    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:27:05.491900 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:05.491877    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:27:06.497791 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:06.497765    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/5.log"
Sep 18 22:27:11.727092 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:11.727059    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:27:11.727601 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:11.727120    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:27:11.727601 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:11.727405    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:27:11.727601 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:11.727594    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:27:17.613390 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:17.613362    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:27:17.613765 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:17.613568    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:27:22.613489 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:22.613463    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:27:22.613866 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:22.613668    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:27:30.614065 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:30.614038    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:27:30.614451 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:30.614191    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:27:35.613964 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:35.613935    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:27:35.614382 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:35.614137    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:27:44.613690 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:44.613614    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:27:44.614158 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:44.613826    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:27:46.614244 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:46.614212    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:27:46.614643 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:46.614428    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:27:55.613247 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:27:55.613221    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:27:55.615690 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:27:55.613450    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:28:01.615757 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:01.615719    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:28:01.616136 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:01.615970    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:28:08.613946 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:08.613905    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:28:08.614372 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:08.614124    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:28:14.308854 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:14.308819    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:28:14.407351 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:14.407320    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:28:16.613472 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:16.613435    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:28:16.613945 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:16.613670    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:28:23.613734 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:23.613701    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:28:23.614103 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:23.613945    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:28:31.616900 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:31.616868    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:28:31.617299 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:31.617040    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:28:36.613677 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:36.613649    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:28:36.921812 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:36.921741    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/9.log"
Sep 18 22:28:36.922214 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:36.922198    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/8.log"
Sep 18 22:28:36.922309 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:36.922234    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901" exitCode=1
Sep 18 22:28:36.922371 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:36.922303    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"}
Sep 18 22:28:36.922435 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:36.922380    2564 scope.go:117] "RemoveContainer" containerID="80839d894a015f14f642baa2eb963ae783a94b7545a16bf8a6e1b841971d2174"
Sep 18 22:28:36.922724 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:36.922708    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:28:36.922958 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:36.922930    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:28:37.928644 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:37.928621    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/9.log"
Sep 18 22:28:44.178659 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:44.178628    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:28:44.179022 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:44.178881    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:28:44.179022 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:44.178986    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:28:44.179180 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:44.179160    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:28:44.613176 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:44.613144    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:28:44.613388 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:44.613369    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:28:44.962056 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:44.961956    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:28:44.962437 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:44.962403    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:28:53.824074 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:53.824042    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:28:53.925908 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:53.925873    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:28:58.613626 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:58.613594    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:28:58.613989 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:58.613777    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:28:59.613333 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:28:59.613299    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:28:59.613579 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:28:59.613555    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:29:11.619075 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:11.619047    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:29:11.619458 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:29:11.619204    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:29:13.613654 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:13.613622    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:29:13.614010 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:29:13.613794    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:29:17.220754 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:17.220721    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:29:17.326437 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:17.326410    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:29:24.614259 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:24.614223    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:29:24.614753 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:29:24.614539    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:29:25.613717 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:25.613684    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:29:25.614030 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:29:25.613882    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:29:39.613534 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:39.613503    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:29:39.613905 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:29:39.613670    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:29:40.613310 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:40.613260    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:29:40.613468 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:29:40.613431    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:29:43.415897 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:43.415858    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:29:43.588792 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:43.588754    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:29:47.802890 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:47.802854    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:29:48.078684 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:48.078571    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:29:51.452015 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:51.451968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:29:53.613794 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:53.613763    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:29:54.304313 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:54.304260    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/6.log"
Sep 18 22:29:54.304743 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:54.304727    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/5.log"
Sep 18 22:29:54.304797 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:54.304761    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad" exitCode=1
Sep 18 22:29:54.304834 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:54.304795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"}
Sep 18 22:29:54.304873 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:54.304844    2564 scope.go:117] "RemoveContainer" containerID="8cbd76b864f32c9279f4e182392bdcea066bd9ae273fb06d2ca8a368e82a85a5"
Sep 18 22:29:54.305161 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:54.305140    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:29:54.305374 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:29:54.305350    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:29:54.614001 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:54.613928    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:29:54.614374 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:29:54.614123    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:29:55.311039 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:55.311011    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/6.log"
Sep 18 22:29:57.345525 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:57.345494    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:29:57.438653 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:29:57.438603    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:30:01.727387 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:01.727346    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:30:01.727877 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:01.727416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:30:01.727877 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:01.727784    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:30:01.728079 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:01.728051    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:30:03.039004 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:03.038973    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:30:07.614033 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:07.614012    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:30:07.614383 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:07.614229    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:30:13.613892 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:13.613863    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:30:13.614324 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:13.614051    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:30:21.615817 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:21.615788    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:30:21.616192 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:21.616070    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:30:21.769787 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:21.769759    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/6.log"
Sep 18 22:30:21.770517 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:21.770499    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/9.log"
Sep 18 22:30:21.774974 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:21.774946    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:30:21.781208 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:21.781187    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/6.log"
Sep 18 22:30:21.781895 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:21.781873    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/9.log"
Sep 18 22:30:21.786872 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:21.786855    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:30:24.613825 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:24.613797    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:30:24.614196 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:24.613997    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:30:35.613642 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:35.613612    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:30:35.614219 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:35.613864    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:30:39.613989 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:39.613958    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:30:39.614463 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:39.614194    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:30:46.613387 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:46.613310    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:30:46.613743 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:46.613497    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:30:51.616216 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:51.616185    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:30:51.616680 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:51.616429    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:30:57.613607 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:30:57.613577    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:30:57.613986 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:30:57.613758    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:31:05.613575 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:05.613539    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:31:05.613959 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:31:05.613780    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:31:10.614076 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:10.614047    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:31:10.614564 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:31:10.614227    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:31:16.613802 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:16.613766    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:31:16.614352 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:31:16.613979    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:31:24.613834 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:24.613805    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:31:24.614323 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:31:24.614006    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:31:28.917135 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:28.917099    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:31:31.615493 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:31.615465    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:31:31.615974 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:31:31.615661    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:31:38.613178 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:38.613152    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:31:38.613565 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:31:38.613359    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:31:46.613535 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:46.613505    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:31:46.613954 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:31:46.613693    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:31:49.614020 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:49.613988    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:31:49.614437 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:31:49.614168    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:31:52.867134 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:52.867093    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:31:52.932139 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:52.932110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:31:58.613723 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:31:58.613692    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:31:58.614084 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:31:58.613883    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:32:03.613768 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:03.613735    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:32:03.614241 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:32:03.613967    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:32:10.613322 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:10.613289    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:32:10.613684 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:32:10.613471    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:32:14.613453 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:14.613376    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:32:14.613800 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:32:14.613553    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:32:24.613560 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:24.613529    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:32:24.613992 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:32:24.613730    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:32:29.613286 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:29.613240    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:32:29.613662 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:32:29.613446    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:32:37.613456 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:37.613416    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:32:37.613920 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:32:37.613589    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:32:42.971045 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:42.971010    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:32:43.060507 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:43.060481    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:32:43.253599 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:43.253527    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:32:43.357418 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:43.357381    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:32:44.613750 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:44.613720    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:32:44.614144 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:32:44.613937    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:32:46.529791 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:46.529757    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:32:49.564991 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:49.564945    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:32:52.613786 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:52.613753    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:32:52.614161 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:32:52.613946    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:32:55.732700 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:55.732596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:32:55.845372 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:55.845337    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:32:57.613783 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:57.613754    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:32:57.614256 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:32:57.613951    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:32:58.409276 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:32:58.409226    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:33:04.614054 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:04.614028    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:33:04.614447 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:04.614229    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:33:09.614205 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:09.614175    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:33:09.614598 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:09.614407    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:33:15.613855 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:15.613819    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:33:15.614310 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:15.614047    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:33:20.613546 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:20.613521    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:33:20.613929 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:20.613688    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:33:29.614079 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:29.614043    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:33:29.614586 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:29.614233    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:33:33.613698 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:33.613662    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:33:33.614100 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:33.613917    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:33:42.613466 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:42.613438    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:33:42.613852 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:42.613640    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:33:47.614357 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:47.614259    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:33:48.421796 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:48.421766    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/10.log"
Sep 18 22:33:48.422227 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:48.422211    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/9.log"
Sep 18 22:33:48.422309 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:48.422245    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d" exitCode=1
Sep 18 22:33:48.422360 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:48.422326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"}
Sep 18 22:33:48.422400 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:48.422386    2564 scope.go:117] "RemoveContainer" containerID="371c1bc7abeb74169ecc57555f02db72a26c3f51905831eac89f1c9f1c7fa901"
Sep 18 22:33:48.422709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:48.422685    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:33:48.422932 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:48.422911    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:33:49.428137 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:49.428110    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/10.log"
Sep 18 22:33:54.178944 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:54.178904    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:33:54.179335 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:54.178973    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:33:54.179335 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:54.179295    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:33:54.179493 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:54.179473    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:33:56.613761 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:33:56.613735    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:33:56.614137 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:33:56.613907    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:34:08.613228 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:08.613195    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:34:08.613775 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:34:08.613387    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:34:08.613775 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:08.613601    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:34:08.613929 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:34:08.613781    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:34:12.887641 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:12.887587    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:34:17.797701 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:17.797618    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:34:17.895186 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:17.895154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:34:21.616123 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:21.616093    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:34:21.616549 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:34:21.616366    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:34:22.846531 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:22.846499    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:34:23.613499 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:23.613474    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:34:23.613665 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:34:23.613643    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:34:30.170711 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:30.170670    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:34:30.455657 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:30.455576    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:34:32.613193 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:32.613165    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:34:32.613590 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:34:32.613378    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:34:36.613874 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:36.613845    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:34:36.614392 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:34:36.614033    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:34:44.536064 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:44.536033    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:34:45.613334 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:45.613304    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:34:45.613710 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:34:45.613527    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:34:46.085975 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:46.085946    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:34:48.262576 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:48.262534    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:34:51.615551 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:51.615524    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:34:51.615946 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:34:51.615704    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:34:52.641160 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:34:52.641128    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:35:00.613420 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:00.613383    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:35:00.766188 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:00.766157    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/7.log"
Sep 18 22:35:00.766699 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:00.766681    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/6.log"
Sep 18 22:35:00.766811 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:00.766719    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9" exitCode=1
Sep 18 22:35:00.766811 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:00.766781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"}
Sep 18 22:35:00.766934 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:00.766838    2564 scope.go:117] "RemoveContainer" containerID="1ccf8e6b47ebcc0d9238024378c9b1491f3e2fccb366d6f5db3bff16f42b74ad"
Sep 18 22:35:00.767203 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:00.767178    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:35:00.767453 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:00.767429    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:35:01.727349 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:01.727311    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:35:01.727729 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:01.727372    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:35:01.773211 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:01.773184    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/7.log"
Sep 18 22:35:01.773589 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:01.773572    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:35:01.773801 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:01.773783    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:35:05.981406 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:05.981369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:35:06.613861 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:06.613829    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:35:06.614063 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:06.614043    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:35:07.873333 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:07.873297    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:35:07.967959 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:07.967927    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:35:16.613302 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:16.613220    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:35:16.613646 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:16.613441    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:35:19.613746 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:19.613718    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:35:19.614097 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:19.613898    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:35:21.799602 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:21.799574    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/7.log"
Sep 18 22:35:21.800411 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:21.800396    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/10.log"
Sep 18 22:35:21.805406 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:21.805385    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:35:21.812165 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:21.812138    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/7.log"
Sep 18 22:35:21.812827 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:21.812811    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/10.log"
Sep 18 22:35:21.817198 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:21.817182    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:35:31.615440 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:31.615413    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:35:31.615818 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:31.615602    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:35:32.613895 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:32.613869    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:35:32.614101 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:32.614082    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:35:44.613347 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:44.613316    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:35:44.613898 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:44.613562    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:35:45.614021 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:45.613990    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:35:45.614412 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:45.614160    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:35:56.613233 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:56.613205    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:35:56.613613 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:56.613383    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:35:57.613488 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:35:57.613461    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:35:57.613851 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:35:57.613643    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:36:05.042749 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:05.042692    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:36:08.613383 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:08.613355    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:36:08.613752 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:36:08.613538    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:36:12.614178 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:12.614140    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:36:12.614581 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:36:12.614418    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:36:20.613353 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:20.613316    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:36:20.613898 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:36:20.613539    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:36:23.613372 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:23.613345    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:36:23.613761 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:36:23.613526    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:36:34.613382 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:34.613343    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:36:34.613877 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:34.613510    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:36:34.613877 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:36:34.613534    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:36:34.613877 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:36:34.613717    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:36:36.026961 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:36.026928    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:36:37.665333 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:37.665300    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:36:45.613581 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:45.613503    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:36:45.614015 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:36:45.613760    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:36:49.613541 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:36:49.613510    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:36:49.613904 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:36:49.613740    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:37:00.613223 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:00.613195    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:37:00.613708 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:00.613409    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:37:03.613197 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:03.613174    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:37:03.613572 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:03.613385    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:37:14.613508 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:14.613479    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:37:14.613909 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:14.613651    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:37:16.613652 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:16.613622    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:37:16.614015 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:16.613798    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:37:25.981249 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:25.981216    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:37:26.086625 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:26.086598    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:37:26.614172 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:26.614148    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:37:26.614369 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:26.614350    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:37:31.615406 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:31.615379    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:37:31.615785 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:31.615579    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:37:32.489786 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:32.489757    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:37:34.327218 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:34.327185    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:37:34.602997 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:34.602915    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:37:38.613633 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:38.613600    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:37:38.613992 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:38.613808    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:37:38.787573 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:38.787542    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:37:38.888530 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:38.888430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:37:42.613208 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:42.613183    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:37:42.613582 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:42.613375    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:37:51.894142 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:51.894108    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:37:53.614458 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:53.614425    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:37:53.614822 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:53.614585    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:37:57.619839 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:37:57.619813    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:37:57.622155 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:37:57.620003    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:38:06.614467 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:06.614437    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:38:06.614865 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:38:06.614584    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:38:09.614080 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:09.614051    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:38:09.614580 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:38:09.614232    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:38:18.613372 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:18.613305    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:38:18.613733 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:38:18.613502    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:38:20.613937 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:20.613907    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:38:20.614332 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:38:20.614079    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:38:29.614066 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:29.614032    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:38:29.614533 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:38:29.614251    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:38:31.607001 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:31.606972    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2_b89dd3a0-382a-4e1a-b6fb-d012198819ef/main/0.log"
Sep 18 22:38:31.615730 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:31.615710    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:38:31.615965 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:38:31.615943    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:38:31.616182 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:31.616167    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/7.log"
Sep 18 22:38:31.623387 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:31.623365    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/7.log"
Sep 18 22:38:32.913925 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:32.913890    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:38:33.021254 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:33.021227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:38:40.452642 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.452606    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"]
Sep 18 22:38:40.456135 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.456118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.459528 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.459501    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 22:38:40.459528 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.459518    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:38:40.459767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.459540    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 22:38:40.459767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.459546    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 18 22:38:40.459767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.459602    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 22:38:40.459767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.459543    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-b9thv\""
Sep 18 22:38:40.467176 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.467153    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"]
Sep 18 22:38:40.478552 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.478531    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:38:40.602218 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.602183    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqq8\" (UniqueName: \"kubernetes.io/projected/9853ed79-cb46-4102-970f-355a479cfdbd-kube-api-access-ggqq8\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.602416 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.602329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-model-cache\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.602416 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.602359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.602513 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.602443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-home\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.602513 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.602465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9853ed79-cb46-4102-970f-355a479cfdbd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.602513 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.602498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-dshm\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.602632 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.602521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9853ed79-cb46-4102-970f-355a479cfdbd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.602632 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.602580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.703536 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.703449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-model-cache\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.703536 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.703483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.703536 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.703523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-home\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.703817 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.703547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9853ed79-cb46-4102-970f-355a479cfdbd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.703817 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.703584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-dshm\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.703817 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.703607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9853ed79-cb46-4102-970f-355a479cfdbd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.703817 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.703633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.703817 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.703660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqq8\" (UniqueName: \"kubernetes.io/projected/9853ed79-cb46-4102-970f-355a479cfdbd-kube-api-access-ggqq8\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.704108 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.703912    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-model-cache\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.704108 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.704059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-home\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.704382 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.704359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.704474 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.704381    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.704474 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.704400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9853ed79-cb46-4102-970f-355a479cfdbd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.706335 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.706309    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9853ed79-cb46-4102-970f-355a479cfdbd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.706459 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.706355    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-dshm\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.714396 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.714367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqq8\" (UniqueName: \"kubernetes.io/projected/9853ed79-cb46-4102-970f-355a479cfdbd-kube-api-access-ggqq8\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.761607 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.761577    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"]
Sep 18 22:38:40.766537 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.766515    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:38:40.766707 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.766679    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:40.770035 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.769873    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-kzcvg\""
Sep 18 22:38:40.780691 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.780666    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"]
Sep 18 22:38:40.789849 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.789822    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:38:40.879840 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.879810    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:38:40.905070 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.905030    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc9290e-ff43-447a-9947-463db63c6a8c-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj\" (UID: \"9bc9290e-ff43-447a-9947-463db63c6a8c\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:38:40.905236 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:40.905154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shz24\" (UniqueName: \"kubernetes.io/projected/9bc9290e-ff43-447a-9947-463db63c6a8c-kube-api-access-shz24\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj\" (UID: \"9bc9290e-ff43-447a-9947-463db63c6a8c\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:38:41.005915 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.005839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shz24\" (UniqueName: \"kubernetes.io/projected/9bc9290e-ff43-447a-9947-463db63c6a8c-kube-api-access-shz24\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj\" (UID: \"9bc9290e-ff43-447a-9947-463db63c6a8c\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:38:41.006043 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.005922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc9290e-ff43-447a-9947-463db63c6a8c-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj\" (UID: \"9bc9290e-ff43-447a-9947-463db63c6a8c\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:38:41.008301 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.008254    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc9290e-ff43-447a-9947-463db63c6a8c-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj\" (UID: \"9bc9290e-ff43-447a-9947-463db63c6a8c\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:38:41.014791 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.014769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shz24\" (UniqueName: \"kubernetes.io/projected/9bc9290e-ff43-447a-9947-463db63c6a8c-kube-api-access-shz24\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj\" (UID: \"9bc9290e-ff43-447a-9947-463db63c6a8c\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:38:41.049896 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.049871    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"]
Sep 18 22:38:41.051594 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:38:41.051560    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9853ed79_cb46_4102_970f_355a479cfdbd.slice/crio-d622815e5028347ec6c9a1208813e52a12aaaaaa0b0861c19c8472b0ebcf8b61 WatchSource:0}: Error finding container d622815e5028347ec6c9a1208813e52a12aaaaaa0b0861c19c8472b0ebcf8b61: Status 404 returned error can't find the container with id d622815e5028347ec6c9a1208813e52a12aaaaaa0b0861c19c8472b0ebcf8b61
Sep 18 22:38:41.053507 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.053492    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:38:41.080934 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.080910    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:38:41.354424 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.354398    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"]
Sep 18 22:38:41.355307 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:38:41.355282    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc9290e_ff43_447a_9947_463db63c6a8c.slice/crio-872a93189936d3ca9d7baadf702be7cc753d2d40146051c01de5168489e37ab6 WatchSource:0}: Error finding container 872a93189936d3ca9d7baadf702be7cc753d2d40146051c01de5168489e37ab6: Status 404 returned error can't find the container with id 872a93189936d3ca9d7baadf702be7cc753d2d40146051c01de5168489e37ab6
Sep 18 22:38:41.813018 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.812876    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" event={"ID":"9bc9290e-ff43-447a-9947-463db63c6a8c","Type":"ContainerStarted","Data":"6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a"}
Sep 18 22:38:41.813018 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.812940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" event={"ID":"9bc9290e-ff43-447a-9947-463db63c6a8c","Type":"ContainerStarted","Data":"872a93189936d3ca9d7baadf702be7cc753d2d40146051c01de5168489e37ab6"}
Sep 18 22:38:41.813553 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.813134    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:38:41.814767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.814706    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" event={"ID":"9853ed79-cb46-4102-970f-355a479cfdbd","Type":"ContainerStarted","Data":"d622815e5028347ec6c9a1208813e52a12aaaaaa0b0861c19c8472b0ebcf8b61"}
Sep 18 22:38:41.832030 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:41.831977    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" podStartSLOduration=1.8319568739999998 podStartE2EDuration="1.831956874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:38:41.831020394 +0000 UTC m=+2900.687676047" watchObservedRunningTime="2026-09-18 22:38:41.831956874 +0000 UTC m=+2900.688612528"
Sep 18 22:38:44.613921 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:44.613886    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:38:44.614500 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:38:44.614127    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:38:44.834006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:44.833971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" event={"ID":"9853ed79-cb46-4102-970f-355a479cfdbd","Type":"ContainerStarted","Data":"6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879"}
Sep 18 22:38:46.614276 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:46.614235    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:38:46.614643 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:38:46.614436    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:38:46.845031 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:46.844997    2564 generic.go:356] "Generic (PLEG): container finished" podID="9853ed79-cb46-4102-970f-355a479cfdbd" containerID="6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879" exitCode=0
Sep 18 22:38:46.845143 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:46.845068    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" event={"ID":"9853ed79-cb46-4102-970f-355a479cfdbd","Type":"ContainerDied","Data":"6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879"}
Sep 18 22:38:47.851967 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:47.851929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" event={"ID":"9853ed79-cb46-4102-970f-355a479cfdbd","Type":"ContainerStarted","Data":"3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb"}
Sep 18 22:38:47.885970 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:47.885921    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" podStartSLOduration=2.1931310489999998 podStartE2EDuration="7.885904847s" totalImagesPullingTime="3.444037732s" totalInitContainerRuntime="2.248736066s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:38:47.883800657 +0000 UTC m=+2906.740456309" watchObservedRunningTime="2026-09-18 22:38:47.885904847 +0000 UTC m=+2906.742560498"
Sep 18 22:38:48.767721 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:48.767689    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:50.767690 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:50.767659    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:50.781006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:50.780980    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:50.876838 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:50.876810    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:38:52.387020 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:52.386984    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:38:59.613912 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:59.613883    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:38:59.916603 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:59.916529    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/11.log"
Sep 18 22:38:59.916981 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:59.916966    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/10.log"
Sep 18 22:38:59.917045 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:59.917000    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85" exitCode=1
Sep 18 22:38:59.917094 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:59.917073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"}
Sep 18 22:38:59.917143 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:59.917133    2564 scope.go:117] "RemoveContainer" containerID="69eb676bcaf91d2892f2dc0343ae8a28781b9b49235f4b56ac8b26e33331874d"
Sep 18 22:38:59.917451 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:38:59.917429    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:38:59.917644 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:38:59.917622    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:39:00.613448 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:00.613420    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:39:00.613639 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:00.613621    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:39:00.923843 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:00.923766    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/11.log"
Sep 18 22:39:04.178905 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:04.178869    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:39:04.179325 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:04.178928    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:39:04.179325 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:04.179292    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:39:04.179531 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:04.179512    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:39:12.823579 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:12.823547    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:39:13.601711 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:13.601679    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:39:13.697121 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:13.697095    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:39:15.613700 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:15.613670    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:39:15.614077 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:15.613860    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:39:18.613712 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:18.613667    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:39:18.614157 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:18.613865    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:39:29.613357 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:29.613325    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:39:29.613747 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:29.613568    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:39:30.613777 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:30.613750    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:39:30.614149 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:30.613948    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:39:32.895346 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:32.895311    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:39:33.006484 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:33.006449    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:39:42.613332 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:42.613298    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:39:42.613719 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:42.613478    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:39:44.614124 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:44.614050    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:39:44.614577 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:44.614229    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:39:55.613579 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:55.613544    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:39:55.613963 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:55.613782    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:39:56.613822 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:39:56.613796    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:39:56.614214 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:39:56.614031    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:40:04.599362 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:04.599327    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"]
Sep 18 22:40:04.608623 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:04.608588    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:40:04.608623 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:40:04.608623 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:40:04.608623 ip-10-0-130-79 kubenswrapper[2564]:  > containerID="3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb" cmd=["/bin/sleep","15"]
Sep 18 22:40:04.608840 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:04.608640    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" message="" err=<
Sep 18 22:40:04.608840 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:40:04.608840 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:40:04.608840 ip-10-0-130-79 kubenswrapper[2564]:  >
Sep 18 22:40:04.608840 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:04.608651    2564 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:40:04.608840 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:40:04.608840 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:40:04.608840 ip-10-0-130-79 kubenswrapper[2564]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" podUID="9853ed79-cb46-4102-970f-355a479cfdbd" containerName="main" containerID="cri-o://3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb"
Sep 18 22:40:04.608840 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:04.608698    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" podUID="9853ed79-cb46-4102-970f-355a479cfdbd" containerName="main" containerID="cri-o://3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb" gracePeriod=60
Sep 18 22:40:04.617350 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:04.617327    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"]
Sep 18 22:40:04.633737 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:04.633705    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:40:04.866538 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:04.866457    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" podUID="9853ed79-cb46-4102-970f-355a479cfdbd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 18 22:40:04.914059 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:04.914030    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:40:04.964925 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:04.964897    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:40:05.064138 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.064101    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggqq8\" (UniqueName: \"kubernetes.io/projected/9853ed79-cb46-4102-970f-355a479cfdbd-kube-api-access-ggqq8\") pod \"9853ed79-cb46-4102-970f-355a479cfdbd\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") "
Sep 18 22:40:05.064365 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.064157    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9853ed79-cb46-4102-970f-355a479cfdbd-cabundle-cert\") pod \"9853ed79-cb46-4102-970f-355a479cfdbd\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") "
Sep 18 22:40:05.064365 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.064190    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9853ed79-cb46-4102-970f-355a479cfdbd-tls-certs\") pod \"9853ed79-cb46-4102-970f-355a479cfdbd\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") "
Sep 18 22:40:05.064541 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.064339    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-dshm\") pod \"9853ed79-cb46-4102-970f-355a479cfdbd\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") "
Sep 18 22:40:05.064732 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.064714    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-home\") pod \"9853ed79-cb46-4102-970f-355a479cfdbd\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") "
Sep 18 22:40:05.064930 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.064737    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9853ed79-cb46-4102-970f-355a479cfdbd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9853ed79-cb46-4102-970f-355a479cfdbd" (UID: "9853ed79-cb46-4102-970f-355a479cfdbd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:40:05.065037 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.065019    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-home" (OuterVolumeSpecName: "home") pod "9853ed79-cb46-4102-970f-355a479cfdbd" (UID: "9853ed79-cb46-4102-970f-355a479cfdbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.065165 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.065047    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-tmp-dir\") pod \"9853ed79-cb46-4102-970f-355a479cfdbd\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") "
Sep 18 22:40:05.065320 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.065302    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9853ed79-cb46-4102-970f-355a479cfdbd" (UID: "9853ed79-cb46-4102-970f-355a479cfdbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.065435 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.065345    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-model-cache\") pod \"9853ed79-cb46-4102-970f-355a479cfdbd\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") "
Sep 18 22:40:05.065583 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.065565    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-model-cache" (OuterVolumeSpecName: "model-cache") pod "9853ed79-cb46-4102-970f-355a479cfdbd" (UID: "9853ed79-cb46-4102-970f-355a479cfdbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.065660 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.065642    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-kserve-provision-location\") pod \"9853ed79-cb46-4102-970f-355a479cfdbd\" (UID: \"9853ed79-cb46-4102-970f-355a479cfdbd\") "
Sep 18 22:40:05.066097 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.066062    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9853ed79-cb46-4102-970f-355a479cfdbd-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.066187 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.066100    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.066187 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.066113    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.066187 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.066125    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.067182 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.067150    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-dshm" (OuterVolumeSpecName: "dshm") pod "9853ed79-cb46-4102-970f-355a479cfdbd" (UID: "9853ed79-cb46-4102-970f-355a479cfdbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.067443 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.067415    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9853ed79-cb46-4102-970f-355a479cfdbd-kube-api-access-ggqq8" (OuterVolumeSpecName: "kube-api-access-ggqq8") pod "9853ed79-cb46-4102-970f-355a479cfdbd" (UID: "9853ed79-cb46-4102-970f-355a479cfdbd"). InnerVolumeSpecName "kube-api-access-ggqq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:40:05.068936 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.068910    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9853ed79-cb46-4102-970f-355a479cfdbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9853ed79-cb46-4102-970f-355a479cfdbd" (UID: "9853ed79-cb46-4102-970f-355a479cfdbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:40:05.087686 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.087655    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9853ed79-cb46-4102-970f-355a479cfdbd" (UID: "9853ed79-cb46-4102-970f-355a479cfdbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.167351 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.167261    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.167351 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.167302    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggqq8\" (UniqueName: \"kubernetes.io/projected/9853ed79-cb46-4102-970f-355a479cfdbd-kube-api-access-ggqq8\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.167351 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.167315    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9853ed79-cb46-4102-970f-355a479cfdbd-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.167351 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.167330    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9853ed79-cb46-4102-970f-355a479cfdbd-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.227952 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.227924    2564 generic.go:356] "Generic (PLEG): container finished" podID="9853ed79-cb46-4102-970f-355a479cfdbd" containerID="3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb" exitCode=0
Sep 18 22:40:05.228098 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.227989    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"
Sep 18 22:40:05.228098 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.228007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" event={"ID":"9853ed79-cb46-4102-970f-355a479cfdbd","Type":"ContainerDied","Data":"3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb"}
Sep 18 22:40:05.228098 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.228072    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j" event={"ID":"9853ed79-cb46-4102-970f-355a479cfdbd","Type":"ContainerDied","Data":"d622815e5028347ec6c9a1208813e52a12aaaaaa0b0861c19c8472b0ebcf8b61"}
Sep 18 22:40:05.228098 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.228094    2564 scope.go:117] "RemoveContainer" containerID="3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb"
Sep 18 22:40:05.237968 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.237952    2564 scope.go:117] "RemoveContainer" containerID="6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879"
Sep 18 22:40:05.245744 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.245725    2564 scope.go:117] "RemoveContainer" containerID="3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb"
Sep 18 22:40:05.245991 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:05.245970    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb\": container with ID starting with 3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb not found: ID does not exist" containerID="3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb"
Sep 18 22:40:05.246061 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.246003    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb"} err="failed to get container status \"3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb\": rpc error: code = NotFound desc = could not find container \"3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb\": container with ID starting with 3b6d6be5e274d4c5cde8d5c44a5e4293cea413de1299b983d78b2c1862eacacb not found: ID does not exist"
Sep 18 22:40:05.246061 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.246028    2564 scope.go:117] "RemoveContainer" containerID="6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879"
Sep 18 22:40:05.246300 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:05.246261    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879\": container with ID starting with 6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879 not found: ID does not exist" containerID="6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879"
Sep 18 22:40:05.246347 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.246311    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879"} err="failed to get container status \"6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879\": rpc error: code = NotFound desc = could not find container \"6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879\": container with ID starting with 6a1bde2fc2096e9a06b356590875ff72af01b42c35b79bb4e876e76eddf55879 not found: ID does not exist"
Sep 18 22:40:05.261261 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.261241    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"]
Sep 18 22:40:05.265817 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.265797    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-bsj8j"]
Sep 18 22:40:05.620373 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:05.620339    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9853ed79-cb46-4102-970f-355a479cfdbd" path="/var/lib/kubelet/pods/9853ed79-cb46-4102-970f-355a479cfdbd/volumes"
Sep 18 22:40:06.613626 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:06.613598    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:40:06.613807 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:06.613789    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:40:11.615461 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:11.615433    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:40:12.061725 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:12.061688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:40:12.177895 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:12.177866    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:40:12.264085 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:12.264014    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/8.log"
Sep 18 22:40:12.264514 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:12.264494    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/7.log"
Sep 18 22:40:12.264634 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:12.264535    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1" exitCode=1
Sep 18 22:40:12.264634 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:12.264608    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"}
Sep 18 22:40:12.264730 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:12.264667    2564 scope.go:117] "RemoveContainer" containerID="ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9"
Sep 18 22:40:12.264981 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:12.264967    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:40:12.265214 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:12.265194    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:40:12.823429 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:12.823397    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" podUID="9bc9290e-ff43-447a-9947-463db63c6a8c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:40:13.271440 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:13.271366    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/8.log"
Sep 18 22:40:18.613821 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:18.613790    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:40:18.614345 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:18.613968    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:40:19.635300 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:19.635249    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" podUID="9bc9290e-ff43-447a-9947-463db63c6a8c" containerName="main" containerID="cri-o://6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a" gracePeriod=45
Sep 18 22:40:21.728073 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:21.728042    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:40:21.728461 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:21.728091    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:40:21.728461 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:21.728396    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:40:21.728617 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:21.728597    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:40:21.831470 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:21.831447    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/8.log"
Sep 18 22:40:21.832161 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:21.832147    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/11.log"
Sep 18 22:40:21.837411 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:21.837391    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:40:21.844194 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:21.844176    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/8.log"
Sep 18 22:40:21.844884 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:21.844868    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/11.log"
Sep 18 22:40:21.849659 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:21.849641    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:40:22.823124 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:22.823089    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" podUID="9bc9290e-ff43-447a-9947-463db63c6a8c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:40:32.614192 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:32.614165    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:40:32.614569 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:32.614341    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:40:32.614569 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:32.614366    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:40:32.614569 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:32.614479    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:40:32.823373 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:32.823336    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" podUID="9bc9290e-ff43-447a-9947-463db63c6a8c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:40:32.823550 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:32.823452    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:40:42.823242 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:42.823204    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" podUID="9bc9290e-ff43-447a-9947-463db63c6a8c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:40:44.613517 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:44.613488    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:40:44.613878 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:44.613656    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:40:45.614254 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:45.614227    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:40:45.614647 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:45.614434    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:40:49.884279 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:49.884238    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:40:49.957416 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:49.957334    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shz24\" (UniqueName: \"kubernetes.io/projected/9bc9290e-ff43-447a-9947-463db63c6a8c-kube-api-access-shz24\") pod \"9bc9290e-ff43-447a-9947-463db63c6a8c\" (UID: \"9bc9290e-ff43-447a-9947-463db63c6a8c\") "
Sep 18 22:40:49.957569 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:49.957440    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc9290e-ff43-447a-9947-463db63c6a8c-tls-certs\") pod \"9bc9290e-ff43-447a-9947-463db63c6a8c\" (UID: \"9bc9290e-ff43-447a-9947-463db63c6a8c\") "
Sep 18 22:40:49.959406 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:49.959380    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc9290e-ff43-447a-9947-463db63c6a8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bc9290e-ff43-447a-9947-463db63c6a8c" (UID: "9bc9290e-ff43-447a-9947-463db63c6a8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:40:49.959530 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:49.959473    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc9290e-ff43-447a-9947-463db63c6a8c-kube-api-access-shz24" (OuterVolumeSpecName: "kube-api-access-shz24") pod "9bc9290e-ff43-447a-9947-463db63c6a8c" (UID: "9bc9290e-ff43-447a-9947-463db63c6a8c"). InnerVolumeSpecName "kube-api-access-shz24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:40:50.058240 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.058202    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc9290e-ff43-447a-9947-463db63c6a8c-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:50.058240 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.058233    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shz24\" (UniqueName: \"kubernetes.io/projected/9bc9290e-ff43-447a-9947-463db63c6a8c-kube-api-access-shz24\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:40:50.444219 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.444183    2564 generic.go:356] "Generic (PLEG): container finished" podID="9bc9290e-ff43-447a-9947-463db63c6a8c" containerID="6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a" exitCode=0
Sep 18 22:40:50.444405 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.444232    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"
Sep 18 22:40:50.444405 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.444282    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" event={"ID":"9bc9290e-ff43-447a-9947-463db63c6a8c","Type":"ContainerDied","Data":"6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a"}
Sep 18 22:40:50.444405 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.444339    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj" event={"ID":"9bc9290e-ff43-447a-9947-463db63c6a8c","Type":"ContainerDied","Data":"872a93189936d3ca9d7baadf702be7cc753d2d40146051c01de5168489e37ab6"}
Sep 18 22:40:50.444405 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.444366    2564 scope.go:117] "RemoveContainer" containerID="6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a"
Sep 18 22:40:50.453870 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.453849    2564 scope.go:117] "RemoveContainer" containerID="6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a"
Sep 18 22:40:50.454131 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:50.454110    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a\": container with ID starting with 6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a not found: ID does not exist" containerID="6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a"
Sep 18 22:40:50.454193 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.454144    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a"} err="failed to get container status \"6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a\": rpc error: code = NotFound desc = could not find container \"6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a\": container with ID starting with 6b383e8636e2135618f92036e052c3c38d9ab48480937765c13c68eb28f3bb6a not found: ID does not exist"
Sep 18 22:40:50.473904 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.473884    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"]
Sep 18 22:40:50.481251 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:50.481231    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-57fcddbf8q4zj"]
Sep 18 22:40:51.618937 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:51.618906    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bc9290e-ff43-447a-9947-463db63c6a8c" path="/var/lib/kubelet/pods/9bc9290e-ff43-447a-9947-463db63c6a8c/volumes"
Sep 18 22:40:53.880499 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:53.880463    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:40:53.984389 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:53.984359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:40:56.613472 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:56.613446    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:40:56.613868 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:56.613630    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:40:59.614256 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:40:59.614231    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:40:59.614653 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:40:59.614459    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:41:00.193326 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.193289    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:00.214510 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.214475    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"]
Sep 18 22:41:00.214896 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.214878    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9853ed79-cb46-4102-970f-355a479cfdbd" containerName="main"
Sep 18 22:41:00.214980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.214898    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9853ed79-cb46-4102-970f-355a479cfdbd" containerName="main"
Sep 18 22:41:00.214980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.214927    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc9290e-ff43-447a-9947-463db63c6a8c" containerName="main"
Sep 18 22:41:00.214980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.214935    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc9290e-ff43-447a-9947-463db63c6a8c" containerName="main"
Sep 18 22:41:00.214980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.214973    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9853ed79-cb46-4102-970f-355a479cfdbd" containerName="storage-initializer"
Sep 18 22:41:00.215188 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.214982    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9853ed79-cb46-4102-970f-355a479cfdbd" containerName="storage-initializer"
Sep 18 22:41:00.215188 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.215088    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9853ed79-cb46-4102-970f-355a479cfdbd" containerName="main"
Sep 18 22:41:00.215188 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.215102    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc9290e-ff43-447a-9947-463db63c6a8c" containerName="main"
Sep 18 22:41:00.219906 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.219887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.222605 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.222583    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 18 22:41:00.222707 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.222626    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-tnktd\""
Sep 18 22:41:00.223304 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.223287    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:41:00.223413 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.223395    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 22:41:00.223468 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.223452    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 22:41:00.223702 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.223688    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 22:41:00.232796 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.232774    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"]
Sep 18 22:41:00.342598 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.342567    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/842f6d8b-add5-4629-af93-a6cf94bc399a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.342733 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.342617    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.342733 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.342665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.342733 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.342728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-home\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.342844 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.342752    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-model-cache\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.342844 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.342791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-dshm\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.342908 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.342839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842f6d8b-add5-4629-af93-a6cf94bc399a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.348977 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.342860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ht6\" (UniqueName: \"kubernetes.io/projected/842f6d8b-add5-4629-af93-a6cf94bc399a-kube-api-access-k8ht6\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.444505 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.444437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.444505 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.444471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.444661 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.444507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-home\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.444661 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.444634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-model-cache\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.444773 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.444688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-dshm\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.444773 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.444727    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842f6d8b-add5-4629-af93-a6cf94bc399a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.444898 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.444874    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.444963 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.444901    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-home\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.445072 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.445042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ht6\" (UniqueName: \"kubernetes.io/projected/842f6d8b-add5-4629-af93-a6cf94bc399a-kube-api-access-k8ht6\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.445189 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.445126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/842f6d8b-add5-4629-af93-a6cf94bc399a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.445189 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.445140    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.445349 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.445325    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-model-cache\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.445867 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.445843    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/842f6d8b-add5-4629-af93-a6cf94bc399a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.447448 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.447426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-dshm\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.447810 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.447790    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842f6d8b-add5-4629-af93-a6cf94bc399a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.456031 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.456005    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ht6\" (UniqueName: \"kubernetes.io/projected/842f6d8b-add5-4629-af93-a6cf94bc399a-kube-api-access-k8ht6\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.526281 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.526241    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:00.530179 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.530163    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:00.654756 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.654724    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:00.833999 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:00.833811    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"]
Sep 18 22:41:00.836655 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:41:00.836624    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842f6d8b_add5_4629_af93_a6cf94bc399a.slice/crio-688db062a90c07ff58f22751036efce7884dbe2b7fe5d0ca79d4f9547cb61ec8 WatchSource:0}: Error finding container 688db062a90c07ff58f22751036efce7884dbe2b7fe5d0ca79d4f9547cb61ec8: Status 404 returned error can't find the container with id 688db062a90c07ff58f22751036efce7884dbe2b7fe5d0ca79d4f9547cb61ec8
Sep 18 22:41:01.502912 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:01.502875    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" event={"ID":"842f6d8b-add5-4629-af93-a6cf94bc399a","Type":"ContainerStarted","Data":"3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a"}
Sep 18 22:41:01.503075 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:01.502932    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" event={"ID":"842f6d8b-add5-4629-af93-a6cf94bc399a","Type":"ContainerStarted","Data":"688db062a90c07ff58f22751036efce7884dbe2b7fe5d0ca79d4f9547cb61ec8"}
Sep 18 22:41:02.154524 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:02.154491    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:02.268811 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:02.268708    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:02.369928 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:02.369893    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:02.466162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:02.466068    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:03.513848 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:03.513821    2564 generic.go:356] "Generic (PLEG): container finished" podID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerID="3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a" exitCode=0
Sep 18 22:41:03.514180 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:03.513854    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" event={"ID":"842f6d8b-add5-4629-af93-a6cf94bc399a","Type":"ContainerDied","Data":"3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a"}
Sep 18 22:41:04.520435 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:04.520390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" event={"ID":"842f6d8b-add5-4629-af93-a6cf94bc399a","Type":"ContainerStarted","Data":"97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f"}
Sep 18 22:41:04.530656 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:04.530631    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:04.550322 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:04.550246    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" podStartSLOduration=2.329258726 podStartE2EDuration="4.550227017s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.220968291s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:41:04.548831646 +0000 UTC m=+3043.405487298" watchObservedRunningTime="2026-09-18 22:41:04.550227017 +0000 UTC m=+3043.406882674"
Sep 18 22:41:08.462390 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:08.462359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:08.559625 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:08.559596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:09.209511 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:09.209478    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:10.530851 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:10.530820    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:10.543239 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:10.543213    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:10.560335 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:10.560314    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:41:10.614061 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:10.614029    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:41:10.614238 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:41:10.614221    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:41:14.613305 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:14.613217    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:41:14.613677 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:41:14.613429    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:41:14.995937 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:14.995852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:15.063694 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:15.063664    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:23.614153 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:23.614125    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:41:23.614568 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:41:23.614319    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:41:28.111345 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:28.111308    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:29.613438 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:29.613412    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:41:29.613800 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:41:29.613582    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:41:31.449541 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:31.449498    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:37.614218 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:37.614186    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:41:37.614625 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:41:37.614420    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:41:41.617341 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:41.617311    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:41:41.617785 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:41:41.617514    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:41:51.615664 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:51.615639    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:41:51.616054 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:41:51.615823    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:41:56.614208 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:56.614179    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:41:56.614677 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:41:56.614377    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:41:58.450406 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.450372    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"]
Sep 18 22:41:58.453882 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.453865    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.457470 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.457440    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 18 22:41:58.457470 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.457441    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 18 22:41:58.457653 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.457637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-6ltmm\""
Sep 18 22:41:58.457719 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.457703    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 18 22:41:58.457962 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.457945    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 18 22:41:58.458185 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.458168    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:41:58.471054 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.470833    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"]
Sep 18 22:41:58.573349 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.573323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-home\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.573494 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.573354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsh97\" (UniqueName: \"kubernetes.io/projected/f4326b5e-43f5-415d-ad75-ea0681731335-kube-api-access-qsh97\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.573494 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.573386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-tmp-dir\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.573494 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.573405    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-dshm\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.573494 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.573421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4326b5e-43f5-415d-ad75-ea0681731335-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.573625 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.573499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.573625 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.573536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-model-cache\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.573625 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.573583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4326b5e-43f5-415d-ad75-ea0681731335-tls-certs\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.576602 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.576579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:41:58.674902 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.674875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-tmp-dir\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675042 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.674907    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-dshm\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675042 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.674932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4326b5e-43f5-415d-ad75-ea0681731335-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675042 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675018    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675204 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-model-cache\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675204 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4326b5e-43f5-415d-ad75-ea0681731335-tls-certs\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675204 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-home\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675204 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsh97\" (UniqueName: \"kubernetes.io/projected/f4326b5e-43f5-415d-ad75-ea0681731335-kube-api-access-qsh97\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675437 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675252    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-tmp-dir\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675437 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675423    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675536 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675452    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-model-cache\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675536 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675489    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-home\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.675711 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.675686    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4326b5e-43f5-415d-ad75-ea0681731335-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.677158 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.677131    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-dshm\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.677555 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.677537    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4326b5e-43f5-415d-ad75-ea0681731335-tls-certs\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.684123 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.684097    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsh97\" (UniqueName: \"kubernetes.io/projected/f4326b5e-43f5-415d-ad75-ea0681731335-kube-api-access-qsh97\") pod \"custom-route-timeout-test-kserve-5f5bff74cc-ks58d\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:58.764986 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:58.764922    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:41:59.047111 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:59.047089    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"]
Sep 18 22:41:59.049648 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:41:59.049618    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4326b5e_43f5_415d_ad75_ea0681731335.slice/crio-c6aa332f812f9a1283235d315a7652e0b4641442cc39b2686a3a58ede9c46e1f WatchSource:0}: Error finding container c6aa332f812f9a1283235d315a7652e0b4641442cc39b2686a3a58ede9c46e1f: Status 404 returned error can't find the container with id c6aa332f812f9a1283235d315a7652e0b4641442cc39b2686a3a58ede9c46e1f
Sep 18 22:41:59.786174 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:59.786135    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" event={"ID":"f4326b5e-43f5-415d-ad75-ea0681731335","Type":"ContainerStarted","Data":"4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1"}
Sep 18 22:41:59.786676 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:59.786195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" event={"ID":"f4326b5e-43f5-415d-ad75-ea0681731335","Type":"ContainerStarted","Data":"c6aa332f812f9a1283235d315a7652e0b4641442cc39b2686a3a58ede9c46e1f"}
Sep 18 22:41:59.948820 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:41:59.948782    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:01.414506 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.414456    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:01.425093 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.425055    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"]
Sep 18 22:42:01.434332 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:01.434299    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:42:01.434332 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:42:01.434332 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:42:01.434332 ip-10-0-130-79 kubenswrapper[2564]:  > containerID="97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f" cmd=["/bin/sleep","15"]
Sep 18 22:42:01.434575 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.434351    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" message="" err=<
Sep 18 22:42:01.434575 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:42:01.434575 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:42:01.434575 ip-10-0-130-79 kubenswrapper[2564]:  >
Sep 18 22:42:01.434575 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:01.434363    2564 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:42:01.434575 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:42:01.434575 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:42:01.434575 ip-10-0-130-79 kubenswrapper[2564]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" podUID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerName="main" containerID="cri-o://97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f"
Sep 18 22:42:01.434575 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.434407    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" podUID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerName="main" containerID="cri-o://97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f" gracePeriod=60
Sep 18 22:42:01.482611 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.482512    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:01.550675 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.550644    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" podUID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 18 22:42:01.689928 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.689904    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:42:01.797861 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.797828    2564 generic.go:356] "Generic (PLEG): container finished" podID="f4326b5e-43f5-415d-ad75-ea0681731335" containerID="4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1" exitCode=0
Sep 18 22:42:01.798037 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.797901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" event={"ID":"f4326b5e-43f5-415d-ad75-ea0681731335","Type":"ContainerDied","Data":"4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1"}
Sep 18 22:42:01.799630 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.799608    2564 generic.go:356] "Generic (PLEG): container finished" podID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerID="97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f" exitCode=0
Sep 18 22:42:01.799756 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.799685    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"
Sep 18 22:42:01.799756 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.799706    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" event={"ID":"842f6d8b-add5-4629-af93-a6cf94bc399a","Type":"ContainerDied","Data":"97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f"}
Sep 18 22:42:01.799852 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.799768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t" event={"ID":"842f6d8b-add5-4629-af93-a6cf94bc399a","Type":"ContainerDied","Data":"688db062a90c07ff58f22751036efce7884dbe2b7fe5d0ca79d4f9547cb61ec8"}
Sep 18 22:42:01.799852 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.799800    2564 scope.go:117] "RemoveContainer" containerID="97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f"
Sep 18 22:42:01.808385 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8ht6\" (UniqueName: \"kubernetes.io/projected/842f6d8b-add5-4629-af93-a6cf94bc399a-kube-api-access-k8ht6\") pod \"842f6d8b-add5-4629-af93-a6cf94bc399a\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") "
Sep 18 22:42:01.808477 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808412    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-tmp-dir\") pod \"842f6d8b-add5-4629-af93-a6cf94bc399a\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") "
Sep 18 22:42:01.808477 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808453    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-dshm\") pod \"842f6d8b-add5-4629-af93-a6cf94bc399a\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") "
Sep 18 22:42:01.808777 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808483    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/842f6d8b-add5-4629-af93-a6cf94bc399a-cabundle-cert\") pod \"842f6d8b-add5-4629-af93-a6cf94bc399a\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") "
Sep 18 22:42:01.808777 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808521    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-model-cache\") pod \"842f6d8b-add5-4629-af93-a6cf94bc399a\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") "
Sep 18 22:42:01.808777 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808548    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-home\") pod \"842f6d8b-add5-4629-af93-a6cf94bc399a\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") "
Sep 18 22:42:01.808777 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808611    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-kserve-provision-location\") pod \"842f6d8b-add5-4629-af93-a6cf94bc399a\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") "
Sep 18 22:42:01.808777 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808638    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842f6d8b-add5-4629-af93-a6cf94bc399a-tls-certs\") pod \"842f6d8b-add5-4629-af93-a6cf94bc399a\" (UID: \"842f6d8b-add5-4629-af93-a6cf94bc399a\") "
Sep 18 22:42:01.808777 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808677    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "842f6d8b-add5-4629-af93-a6cf94bc399a" (UID: "842f6d8b-add5-4629-af93-a6cf94bc399a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:01.809073 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808861    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842f6d8b-add5-4629-af93-a6cf94bc399a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "842f6d8b-add5-4629-af93-a6cf94bc399a" (UID: "842f6d8b-add5-4629-af93-a6cf94bc399a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:42:01.809073 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.808959    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-home" (OuterVolumeSpecName: "home") pod "842f6d8b-add5-4629-af93-a6cf94bc399a" (UID: "842f6d8b-add5-4629-af93-a6cf94bc399a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:01.809073 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.809034    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-model-cache" (OuterVolumeSpecName: "model-cache") pod "842f6d8b-add5-4629-af93-a6cf94bc399a" (UID: "842f6d8b-add5-4629-af93-a6cf94bc399a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:01.809305 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.809083    2564 scope.go:117] "RemoveContainer" containerID="3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a"
Sep 18 22:42:01.809305 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.809094    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:42:01.809305 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.809115    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/842f6d8b-add5-4629-af93-a6cf94bc399a-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:42:01.811119 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.811096    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-dshm" (OuterVolumeSpecName: "dshm") pod "842f6d8b-add5-4629-af93-a6cf94bc399a" (UID: "842f6d8b-add5-4629-af93-a6cf94bc399a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:01.811215 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.811130    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842f6d8b-add5-4629-af93-a6cf94bc399a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "842f6d8b-add5-4629-af93-a6cf94bc399a" (UID: "842f6d8b-add5-4629-af93-a6cf94bc399a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:42:01.811215 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.811143    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842f6d8b-add5-4629-af93-a6cf94bc399a-kube-api-access-k8ht6" (OuterVolumeSpecName: "kube-api-access-k8ht6") pod "842f6d8b-add5-4629-af93-a6cf94bc399a" (UID: "842f6d8b-add5-4629-af93-a6cf94bc399a"). InnerVolumeSpecName "kube-api-access-k8ht6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:42:01.823846 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.823818    2564 scope.go:117] "RemoveContainer" containerID="97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f"
Sep 18 22:42:01.824167 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:01.824141    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f\": container with ID starting with 97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f not found: ID does not exist" containerID="97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f"
Sep 18 22:42:01.824224 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.824178    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f"} err="failed to get container status \"97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f\": rpc error: code = NotFound desc = could not find container \"97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f\": container with ID starting with 97025c7a8f56a4615306bff74f22e09d9094ed34b6d0bc87e6d3c169bd590a1f not found: ID does not exist"
Sep 18 22:42:01.824224 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.824202    2564 scope.go:117] "RemoveContainer" containerID="3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a"
Sep 18 22:42:01.824514 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:01.824496    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a\": container with ID starting with 3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a not found: ID does not exist" containerID="3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a"
Sep 18 22:42:01.824566 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.824521    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a"} err="failed to get container status \"3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a\": rpc error: code = NotFound desc = could not find container \"3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a\": container with ID starting with 3b5390bdb161b798e1de522dccbe3331efaa98f6c2c2a9b0e87c0c24c213878a not found: ID does not exist"
Sep 18 22:42:01.828353 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.828331    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "842f6d8b-add5-4629-af93-a6cf94bc399a" (UID: "842f6d8b-add5-4629-af93-a6cf94bc399a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:01.909816 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.909787    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:42:01.909816 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.909815    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:42:01.909965 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.909824    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:42:01.909965 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.909833    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/842f6d8b-add5-4629-af93-a6cf94bc399a-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:42:01.909965 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.909845    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842f6d8b-add5-4629-af93-a6cf94bc399a-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:42:01.909965 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:01.909857    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ht6\" (UniqueName: \"kubernetes.io/projected/842f6d8b-add5-4629-af93-a6cf94bc399a-kube-api-access-k8ht6\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.121749 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:02.121723    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"]
Sep 18 22:42:02.125942 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:02.125918    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-8px9t"]
Sep 18 22:42:03.614010 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:03.613798    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:42:03.614463 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:03.614069    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:42:03.629283 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:03.628953    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="842f6d8b-add5-4629-af93-a6cf94bc399a" path="/var/lib/kubelet/pods/842f6d8b-add5-4629-af93-a6cf94bc399a/volumes"
Sep 18 22:42:11.318797 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:11.318763    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:11.621330 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:11.621248    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:42:11.621521 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:11.621497    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:42:18.613813 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:18.613778    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:42:18.614406 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:18.614038    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:42:26.614251 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:26.614221    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:42:26.614703 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:26.614522    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:42:29.982328 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:29.982297    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" event={"ID":"f4326b5e-43f5-415d-ad75-ea0681731335","Type":"ContainerStarted","Data":"93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0"}
Sep 18 22:42:30.011931 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:30.011853    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podStartSLOduration=2.652903309 podStartE2EDuration="32.011810802s" totalImagesPullingTime="27.247983134s" totalInitContainerRuntime="2.110924359s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:42:30.005474188 +0000 UTC m=+3128.862129866" watchObservedRunningTime="2026-09-18 22:42:30.011810802 +0000 UTC m=+3128.868466459"
Sep 18 22:42:30.765062 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:30.765031    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:42:33.613689 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:33.613652    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:42:33.614088 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:33.613886    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:42:37.614203 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:37.614168    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:42:37.614617 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:37.614405    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:42:38.765667 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:38.765629    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:42:38.767104 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:38.767075    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:42:46.613711 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:46.613610    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:42:46.614132 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:46.613917    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:42:46.934738 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:46.934641    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:48.613467 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:48.613439    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:42:48.614536 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:48.614305    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:42:48.766211 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:48.766167    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:42:49.700989 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:49.700956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:49.794296 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:49.794241    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:56.052767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:56.052734    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:58.136644 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.136614    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"]
Sep 18 22:42:58.139277 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.137117    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerName="storage-initializer"
Sep 18 22:42:58.139277 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.137133    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerName="storage-initializer"
Sep 18 22:42:58.139277 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.137143    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerName="main"
Sep 18 22:42:58.139277 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.137150    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerName="main"
Sep 18 22:42:58.139277 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.137215    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="842f6d8b-add5-4629-af93-a6cf94bc399a" containerName="main"
Sep 18 22:42:58.140496 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.140471    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.142993 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.142945    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 22:42:58.143162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.143133    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-7wfdb\""
Sep 18 22:42:58.143260 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.143174    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:42:58.144040 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.144024    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 22:42:58.144136 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.144031    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 18 22:42:58.144136 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.144057    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 22:42:58.150982 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.150956    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"]
Sep 18 22:42:58.152593 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.152570    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:58.258333 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.258290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzxvg\" (UniqueName: \"kubernetes.io/projected/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kube-api-access-gzxvg\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.258494 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.258375    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-dshm\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.258494 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.258408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tmp-dir\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.258494 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.258469    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-model-cache\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.258598 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.258494    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.258598 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.258520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tls-certs\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.258598 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.258560    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.258698 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.258659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-home\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.360077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.360041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-home\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.360299 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.360132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzxvg\" (UniqueName: \"kubernetes.io/projected/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kube-api-access-gzxvg\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.360299 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.360177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-dshm\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.360299 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.360208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tmp-dir\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.360481 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.360301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-model-cache\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.360481 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.360327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.360481 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.360355    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tls-certs\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.360481 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.360402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.360692 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.360579    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-home\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.361441 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.361414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.361578 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.361522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-model-cache\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.361650 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.361619    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tmp-dir\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.362000 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.361978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.363788 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.363770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tls-certs\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.363877 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.363771    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-dshm\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.377652 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.377626    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzxvg\" (UniqueName: \"kubernetes.io/projected/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kube-api-access-gzxvg\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-w6c6f\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.438003 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.437924    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:58.452073 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.452033    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:42:58.543864 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.543827    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:42:58.615065 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.615033    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:42:58.615312 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:42:58.615290    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:42:58.740807 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.740779    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"]
Sep 18 22:42:58.742942 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:42:58.742910    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f5f954_6ad3_461a_96b6_a9e8e5a45d71.slice/crio-88335f1603d4fb6e821c04584dea9b91e1e997cfa0613acd1981cca2a8120405 WatchSource:0}: Error finding container 88335f1603d4fb6e821c04584dea9b91e1e997cfa0613acd1981cca2a8120405: Status 404 returned error can't find the container with id 88335f1603d4fb6e821c04584dea9b91e1e997cfa0613acd1981cca2a8120405
Sep 18 22:42:58.766163 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:58.766132    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:42:59.147594 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:59.147556    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" event={"ID":"36f5f954-6ad3-461a-96b6-a9e8e5a45d71","Type":"ContainerStarted","Data":"e7ba4d2121ee8199841cba93aabbbec2f50ceff5752838dbc6ada2c2ea682166"}
Sep 18 22:42:59.147989 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:42:59.147621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" event={"ID":"36f5f954-6ad3-461a-96b6-a9e8e5a45d71","Type":"ContainerStarted","Data":"88335f1603d4fb6e821c04584dea9b91e1e997cfa0613acd1981cca2a8120405"}
Sep 18 22:43:00.952199 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:00.952162    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:43:01.070046 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:01.070019    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:43:02.168138 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:02.168101    2564 generic.go:356] "Generic (PLEG): container finished" podID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerID="e7ba4d2121ee8199841cba93aabbbec2f50ceff5752838dbc6ada2c2ea682166" exitCode=0
Sep 18 22:43:02.168532 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:02.168150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" event={"ID":"36f5f954-6ad3-461a-96b6-a9e8e5a45d71","Type":"ContainerDied","Data":"e7ba4d2121ee8199841cba93aabbbec2f50ceff5752838dbc6ada2c2ea682166"}
Sep 18 22:43:03.177327 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:03.177287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" event={"ID":"36f5f954-6ad3-461a-96b6-a9e8e5a45d71","Type":"ContainerStarted","Data":"a8dba567ea3459dc7b863a209b50302cc519d5b75dd5a77f8e0eda51cf82068c"}
Sep 18 22:43:03.199550 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:03.199500    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" podStartSLOduration=2.7561308159999998 podStartE2EDuration="5.199484117s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.443353301s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:43:03.19845544 +0000 UTC m=+3162.055111112" watchObservedRunningTime="2026-09-18 22:43:03.199484117 +0000 UTC m=+3162.056139803"
Sep 18 22:43:03.452402 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:03.452323    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:43:03.620891 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:03.620860    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:43:03.621144 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:03.621109    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:43:08.453223 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:08.453180    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:43:08.465871 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:08.465851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:43:08.766343 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:08.766242    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:43:09.216925 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:09.216896    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:43:11.428397 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:11.428360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:43:13.613312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:13.613278    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:43:13.613767 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:13.613535    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:43:17.619487 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:17.619460    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:43:17.619834 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:17.619647    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:43:18.765634 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:18.765594    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:43:24.613827 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:24.613792    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:43:24.614222 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:24.614047    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:43:28.765498 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:28.765447    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:43:31.616797 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:31.616770    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:43:31.617171 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:31.616964    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:43:35.621467 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:35.621440    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:43:35.621857 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:35.621634    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:43:36.061708 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.061670    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"]
Sep 18 22:43:36.072443 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.072404    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:43:36.072682 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:36.072643    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:43:36.072682 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:43:36.072682 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:43:36.072682 ip-10-0-130-79 kubenswrapper[2564]:  > containerID="a8dba567ea3459dc7b863a209b50302cc519d5b75dd5a77f8e0eda51cf82068c" cmd=["/bin/sleep","15"]
Sep 18 22:43:36.072920 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.072708    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" message="" err=<
Sep 18 22:43:36.072920 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:43:36.072920 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:43:36.072920 ip-10-0-130-79 kubenswrapper[2564]:  >
Sep 18 22:43:36.072920 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:36.072722    2564 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:43:36.072920 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:43:36.072920 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:43:36.072920 ip-10-0-130-79 kubenswrapper[2564]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" podUID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerName="main" containerID="cri-o://a8dba567ea3459dc7b863a209b50302cc519d5b75dd5a77f8e0eda51cf82068c"
Sep 18 22:43:36.072920 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.072763    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" podUID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerName="main" containerID="cri-o://a8dba567ea3459dc7b863a209b50302cc519d5b75dd5a77f8e0eda51cf82068c" gracePeriod=60
Sep 18 22:43:36.174997 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.174967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:43:36.207168 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.207131    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" podUID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 18 22:43:36.352514 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.352479    2564 generic.go:356] "Generic (PLEG): container finished" podID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerID="a8dba567ea3459dc7b863a209b50302cc519d5b75dd5a77f8e0eda51cf82068c" exitCode=0
Sep 18 22:43:36.352651 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.352520    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" event={"ID":"36f5f954-6ad3-461a-96b6-a9e8e5a45d71","Type":"ContainerDied","Data":"a8dba567ea3459dc7b863a209b50302cc519d5b75dd5a77f8e0eda51cf82068c"}
Sep 18 22:43:36.416933 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.416907    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:43:36.544252 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544219    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-cabundle-cert\") pod \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") "
Sep 18 22:43:36.544464 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544278    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-home\") pod \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") "
Sep 18 22:43:36.544464 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544303    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzxvg\" (UniqueName: \"kubernetes.io/projected/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kube-api-access-gzxvg\") pod \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") "
Sep 18 22:43:36.544464 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544345    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kserve-provision-location\") pod \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") "
Sep 18 22:43:36.544464 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544445    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-dshm\") pod \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") "
Sep 18 22:43:36.544693 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544470    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tmp-dir\") pod \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") "
Sep 18 22:43:36.544693 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544504    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-model-cache\") pod \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") "
Sep 18 22:43:36.544693 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544529    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tls-certs\") pod \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\" (UID: \"36f5f954-6ad3-461a-96b6-a9e8e5a45d71\") "
Sep 18 22:43:36.544693 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544606    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "36f5f954-6ad3-461a-96b6-a9e8e5a45d71" (UID: "36f5f954-6ad3-461a-96b6-a9e8e5a45d71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:36.544906 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544802    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36f5f954-6ad3-461a-96b6-a9e8e5a45d71" (UID: "36f5f954-6ad3-461a-96b6-a9e8e5a45d71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.544906 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544816    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-model-cache" (OuterVolumeSpecName: "model-cache") pod "36f5f954-6ad3-461a-96b6-a9e8e5a45d71" (UID: "36f5f954-6ad3-461a-96b6-a9e8e5a45d71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.544906 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544862    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-home" (OuterVolumeSpecName: "home") pod "36f5f954-6ad3-461a-96b6-a9e8e5a45d71" (UID: "36f5f954-6ad3-461a-96b6-a9e8e5a45d71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.545054 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544915    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.545054 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544936    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.545054 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.544952    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.546709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.546679    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36f5f954-6ad3-461a-96b6-a9e8e5a45d71" (UID: "36f5f954-6ad3-461a-96b6-a9e8e5a45d71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:43:36.546709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.546690    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kube-api-access-gzxvg" (OuterVolumeSpecName: "kube-api-access-gzxvg") pod "36f5f954-6ad3-461a-96b6-a9e8e5a45d71" (UID: "36f5f954-6ad3-461a-96b6-a9e8e5a45d71"). InnerVolumeSpecName "kube-api-access-gzxvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:36.546948 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.546925    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-dshm" (OuterVolumeSpecName: "dshm") pod "36f5f954-6ad3-461a-96b6-a9e8e5a45d71" (UID: "36f5f954-6ad3-461a-96b6-a9e8e5a45d71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.563789 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.563764    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36f5f954-6ad3-461a-96b6-a9e8e5a45d71" (UID: "36f5f954-6ad3-461a-96b6-a9e8e5a45d71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.646206 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.646128    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.646206 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.646154    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.646206 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.646164    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.646206 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.646173    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzxvg\" (UniqueName: \"kubernetes.io/projected/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kube-api-access-gzxvg\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.646206 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:36.646182    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36f5f954-6ad3-461a-96b6-a9e8e5a45d71-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:43:37.360538 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:37.360505    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"
Sep 18 22:43:37.360711 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:37.360506    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f" event={"ID":"36f5f954-6ad3-461a-96b6-a9e8e5a45d71","Type":"ContainerDied","Data":"88335f1603d4fb6e821c04584dea9b91e1e997cfa0613acd1981cca2a8120405"}
Sep 18 22:43:37.360711 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:37.360644    2564 scope.go:117] "RemoveContainer" containerID="a8dba567ea3459dc7b863a209b50302cc519d5b75dd5a77f8e0eda51cf82068c"
Sep 18 22:43:37.370945 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:37.370925    2564 scope.go:117] "RemoveContainer" containerID="e7ba4d2121ee8199841cba93aabbbec2f50ceff5752838dbc6ada2c2ea682166"
Sep 18 22:43:37.382354 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:37.382330    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"]
Sep 18 22:43:37.389414 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:37.389390    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-w6c6f"]
Sep 18 22:43:37.621774 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:37.621698    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" path="/var/lib/kubelet/pods/36f5f954-6ad3-461a-96b6-a9e8e5a45d71/volumes"
Sep 18 22:43:38.765405 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:38.765364    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:43:42.613468 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:42.613438    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:43:42.613880 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:42.613625    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:43:48.613634 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:48.613600    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:43:48.614140 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:48.613830    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:43:48.765720 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:48.765676    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:43:50.654152 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:50.654114    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:43:57.613635 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:57.613606    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:43:57.614117 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:43:57.613798    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:43:58.766020 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:43:58.765966    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:44:01.615872 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:01.615842    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:44:01.616495 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:01.616024    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:44:08.766183 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:08.766136    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:44:11.616739 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:11.616705    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:44:12.546145 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:12.546113    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/12.log"
Sep 18 22:44:12.546575 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:12.546556    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/11.log"
Sep 18 22:44:12.546690 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:12.546599    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277" exitCode=1
Sep 18 22:44:12.546750 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:12.546679    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"}
Sep 18 22:44:12.546750 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:12.546744    2564 scope.go:117] "RemoveContainer" containerID="0d53c05b9bc0f1d7d5302a91500aa7dda213136600a5364c20f92eb784fe9e85"
Sep 18 22:44:12.547101 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:12.547075    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:44:12.547352 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:12.547328    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:44:13.553799 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:13.553769    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/12.log"
Sep 18 22:44:14.178926 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:14.178887    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:44:14.179101 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:14.178955    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:44:14.179286 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:14.179257    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:44:14.179496 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:14.179479    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:44:15.613554 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:15.613527    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:44:15.613923 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:15.613747    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:44:18.775146 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:18.775118    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:44:18.782730 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:18.782702    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:44:23.271677 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:23.271645    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"]
Sep 18 22:44:23.275847 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:23.275814    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:23.337770 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:23.337739    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:25.800776 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:25.800738    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:25.903488 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:25.903444    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:27.613986 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:27.613907    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:44:27.614396 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:27.614109    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:44:29.613644 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:29.613617    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:44:29.614035 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:29.613791    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:44:32.343666 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:32.343620    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:34.147822 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:34.147768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:34.334862 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:34.334831    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:34.998467 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:34.998433    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:38.296460 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:38.296421    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" containerID="cri-o://93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0" gracePeriod=45
Sep 18 22:44:38.569448 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:38.569346    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:38.663879 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:38.663845    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:38.776484 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:38.776436    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 18 22:44:39.048980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.048959    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:44:39.140534 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140460    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-home\") pod \"f4326b5e-43f5-415d-ad75-ea0681731335\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") "
Sep 18 22:44:39.140534 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140506    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4326b5e-43f5-415d-ad75-ea0681731335-cabundle-cert\") pod \"f4326b5e-43f5-415d-ad75-ea0681731335\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") "
Sep 18 22:44:39.140534 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140530    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-model-cache\") pod \"f4326b5e-43f5-415d-ad75-ea0681731335\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") "
Sep 18 22:44:39.140816 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140644    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsh97\" (UniqueName: \"kubernetes.io/projected/f4326b5e-43f5-415d-ad75-ea0681731335-kube-api-access-qsh97\") pod \"f4326b5e-43f5-415d-ad75-ea0681731335\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") "
Sep 18 22:44:39.140816 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140680    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4326b5e-43f5-415d-ad75-ea0681731335-tls-certs\") pod \"f4326b5e-43f5-415d-ad75-ea0681731335\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") "
Sep 18 22:44:39.140816 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140706    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-tmp-dir\") pod \"f4326b5e-43f5-415d-ad75-ea0681731335\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") "
Sep 18 22:44:39.140816 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140758    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-dshm\") pod \"f4326b5e-43f5-415d-ad75-ea0681731335\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") "
Sep 18 22:44:39.140816 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140774    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4326b5e-43f5-415d-ad75-ea0681731335" (UID: "f4326b5e-43f5-415d-ad75-ea0681731335"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:44:39.140816 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140792    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-kserve-provision-location\") pod \"f4326b5e-43f5-415d-ad75-ea0681731335\" (UID: \"f4326b5e-43f5-415d-ad75-ea0681731335\") "
Sep 18 22:44:39.141136 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.140937    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4326b5e-43f5-415d-ad75-ea0681731335-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4326b5e-43f5-415d-ad75-ea0681731335" (UID: "f4326b5e-43f5-415d-ad75-ea0681731335"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:44:39.141250 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.141231    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4326b5e-43f5-415d-ad75-ea0681731335-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:44:39.141340 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.141262    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:44:39.141340 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.141233    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-home" (OuterVolumeSpecName: "home") pod "f4326b5e-43f5-415d-ad75-ea0681731335" (UID: "f4326b5e-43f5-415d-ad75-ea0681731335"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:44:39.142939 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.142913    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-dshm" (OuterVolumeSpecName: "dshm") pod "f4326b5e-43f5-415d-ad75-ea0681731335" (UID: "f4326b5e-43f5-415d-ad75-ea0681731335"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:44:39.143224 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.143190    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4326b5e-43f5-415d-ad75-ea0681731335-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4326b5e-43f5-415d-ad75-ea0681731335" (UID: "f4326b5e-43f5-415d-ad75-ea0681731335"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:44:39.143353 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.143253    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4326b5e-43f5-415d-ad75-ea0681731335-kube-api-access-qsh97" (OuterVolumeSpecName: "kube-api-access-qsh97") pod "f4326b5e-43f5-415d-ad75-ea0681731335" (UID: "f4326b5e-43f5-415d-ad75-ea0681731335"). InnerVolumeSpecName "kube-api-access-qsh97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:44:39.153312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.153290    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4326b5e-43f5-415d-ad75-ea0681731335" (UID: "f4326b5e-43f5-415d-ad75-ea0681731335"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:44:39.161181 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.161158    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4326b5e-43f5-415d-ad75-ea0681731335" (UID: "f4326b5e-43f5-415d-ad75-ea0681731335"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:44:39.241985 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.241964    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:44:39.242087 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.241990    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsh97\" (UniqueName: \"kubernetes.io/projected/f4326b5e-43f5-415d-ad75-ea0681731335-kube-api-access-qsh97\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:44:39.242087 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.242002    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4326b5e-43f5-415d-ad75-ea0681731335-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:44:39.242087 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.242011    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:44:39.242087 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.242018    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:44:39.242087 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.242026    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4326b5e-43f5-415d-ad75-ea0681731335-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:44:39.681775 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.681740    2564 generic.go:356] "Generic (PLEG): container finished" podID="f4326b5e-43f5-415d-ad75-ea0681731335" containerID="93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0" exitCode=0
Sep 18 22:44:39.682116 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.681797    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"
Sep 18 22:44:39.682116 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.681820    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" event={"ID":"f4326b5e-43f5-415d-ad75-ea0681731335","Type":"ContainerDied","Data":"93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0"}
Sep 18 22:44:39.682116 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.681880    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d" event={"ID":"f4326b5e-43f5-415d-ad75-ea0681731335","Type":"ContainerDied","Data":"c6aa332f812f9a1283235d315a7652e0b4641442cc39b2686a3a58ede9c46e1f"}
Sep 18 22:44:39.682116 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.681903    2564 scope.go:117] "RemoveContainer" containerID="93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0"
Sep 18 22:44:39.690289 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.690255    2564 scope.go:117] "RemoveContainer" containerID="4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1"
Sep 18 22:44:39.697676 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.697660    2564 scope.go:117] "RemoveContainer" containerID="93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0"
Sep 18 22:44:39.697955 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:39.697933    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0\": container with ID starting with 93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0 not found: ID does not exist" containerID="93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0"
Sep 18 22:44:39.698052 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.697968    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0"} err="failed to get container status \"93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0\": rpc error: code = NotFound desc = could not find container \"93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0\": container with ID starting with 93dcbf954a494735fea3d86146451000d93265eebb42af0fdec4cb95acbc51f0 not found: ID does not exist"
Sep 18 22:44:39.698052 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.697993    2564 scope.go:117] "RemoveContainer" containerID="4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1"
Sep 18 22:44:39.698466 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:39.698443    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1\": container with ID starting with 4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1 not found: ID does not exist" containerID="4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1"
Sep 18 22:44:39.698572 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.698473    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1"} err="failed to get container status \"4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1\": rpc error: code = NotFound desc = could not find container \"4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1\": container with ID starting with 4170001e44b253e9e91f5ecb75a08bc9ca25916b2951c2bd5ca2955768d40fa1 not found: ID does not exist"
Sep 18 22:44:39.703425 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.703403    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"]
Sep 18 22:44:39.707560 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:39.707540    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f5bff74cc-ks58d"]
Sep 18 22:44:40.614014 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:40.613987    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:44:40.614225 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:40.614148    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:44:41.618970 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:41.618942    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" path="/var/lib/kubelet/pods/f4326b5e-43f5-415d-ad75-ea0681731335/volumes"
Sep 18 22:44:44.613751 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:44.613721    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:44:44.614221 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:44.613931    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:44:47.408477 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:47.408442    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:44:54.614429 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:54.614393    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:44:54.614910 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:54.614635    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:44:55.613558 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:44:55.613526    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:44:55.613742 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:44:55.613711    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:45:07.613644 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:07.613609    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:45:07.614123 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:45:07.613821    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:45:09.469922 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:09.469885    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:09.569415 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:09.569380    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:10.613981 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:10.613952    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:45:10.614399 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:45:10.614171    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:45:13.360181 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:13.360147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:13.452078 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:13.452039    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:13.629226 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:13.629135    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:13.703547 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:13.703507    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:16.703905 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:16.703842    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:19.614112 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:19.614077    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:45:19.614541 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:45:19.614298    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:45:20.269885 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:20.269849    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:21.862782 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:21.862752    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/8.log"
Sep 18 22:45:21.863602 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:21.863582    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/12.log"
Sep 18 22:45:21.868793 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:21.868769    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:45:21.876727 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:21.876704    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/8.log"
Sep 18 22:45:21.877437 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:21.877419    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/12.log"
Sep 18 22:45:21.882371 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:21.882348    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:45:22.613783 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:22.613753    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:45:22.898877 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:22.898798    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/9.log"
Sep 18 22:45:22.899315 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:22.899300    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/8.log"
Sep 18 22:45:22.899371 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:22.899336    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4" exitCode=1
Sep 18 22:45:22.899415 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:22.899393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"}
Sep 18 22:45:22.899468 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:22.899458    2564 scope.go:117] "RemoveContainer" containerID="4fde8789c7135be188741f37f1df172af3a7b7049c4df9e96432da03bdf430c1"
Sep 18 22:45:22.899846 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:22.899829    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:45:22.900068 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:45:22.900047    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:45:23.906025 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:23.905996    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/9.log"
Sep 18 22:45:26.135328 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:26.135292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:26.221800 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:26.221764    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:31.727223 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:31.727184    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:45:31.727658 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:31.727248    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:45:31.727658 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:31.727598    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:45:31.727801 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:45:31.727783    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:45:33.614187 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:33.614157    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:45:33.614645 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:45:33.614384    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:45:38.352709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:38.352672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:38.709690 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:38.709606    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:38.810483 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:38.810447    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:45.613542 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:45.613460    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:45:45.613957 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:45.613545    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:45:45.613957 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:45:45.613676    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:45:45.613957 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:45:45.613695    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:45:56.613713 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:56.613679    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:45:56.614106 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:45:56.613849    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:45:59.526242 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:59.526202    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:45:59.627945 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:45:59.627916    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:46:00.613654 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:00.613627    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:46:00.614048 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:46:00.613853    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:46:05.672602 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:05.672567    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:46:08.613353 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:08.613322    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:46:08.613746 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:46:08.613514    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:46:11.610554 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:11.610515    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:46:11.685772 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:11.685736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:46:14.614096 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:14.614062    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:46:14.614580 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:46:14.614353    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:46:21.854746 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:21.854702    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:46:22.029418 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029384    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"]
Sep 18 22:46:22.029821 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029801    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main"
Sep 18 22:46:22.029821 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029819    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main"
Sep 18 22:46:22.030006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029830    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="storage-initializer"
Sep 18 22:46:22.030006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029837    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="storage-initializer"
Sep 18 22:46:22.030006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029859    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerName="main"
Sep 18 22:46:22.030006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029866    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerName="main"
Sep 18 22:46:22.030006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029877    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerName="storage-initializer"
Sep 18 22:46:22.030006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029883    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerName="storage-initializer"
Sep 18 22:46:22.030006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029947    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4326b5e-43f5-415d-ad75-ea0681731335" containerName="main"
Sep 18 22:46:22.030006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.029957    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="36f5f954-6ad3-461a-96b6-a9e8e5a45d71" containerName="main"
Sep 18 22:46:22.033113 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.033097    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:46:22.038232 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.038208    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-pcgsz\""
Sep 18 22:46:22.038560 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.038535    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 18 22:46:22.038560 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.038546    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-gqrsv\""
Sep 18 22:46:22.038945 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.038571    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls5sz\" (UniqueName: \"kubernetes.io/projected/43e91c78-e5e9-4503-aa5c-3b5851571269-kube-api-access-ls5sz\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f\" (UID: \"43e91c78-e5e9-4503-aa5c-3b5851571269\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:46:22.038945 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.038594    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 18 22:46:22.038945 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.038615    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 18 22:46:22.038945 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.038634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e91c78-e5e9-4503-aa5c-3b5851571269-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f\" (UID: \"43e91c78-e5e9-4503-aa5c-3b5851571269\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:46:22.057591 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.057556    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"]
Sep 18 22:46:22.085066 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.085031    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:46:22.139395 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.139306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls5sz\" (UniqueName: \"kubernetes.io/projected/43e91c78-e5e9-4503-aa5c-3b5851571269-kube-api-access-ls5sz\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f\" (UID: \"43e91c78-e5e9-4503-aa5c-3b5851571269\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:46:22.139395 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.139356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e91c78-e5e9-4503-aa5c-3b5851571269-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f\" (UID: \"43e91c78-e5e9-4503-aa5c-3b5851571269\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:46:22.142023 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.142001    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e91c78-e5e9-4503-aa5c-3b5851571269-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f\" (UID: \"43e91c78-e5e9-4503-aa5c-3b5851571269\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:46:22.148570 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.148543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls5sz\" (UniqueName: \"kubernetes.io/projected/43e91c78-e5e9-4503-aa5c-3b5851571269-kube-api-access-ls5sz\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f\" (UID: \"43e91c78-e5e9-4503-aa5c-3b5851571269\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:46:22.343708 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.343671    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:46:22.620620 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:22.620588    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"]
Sep 18 22:46:22.623252 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:46:22.623227    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e91c78_e5e9_4503_aa5c_3b5851571269.slice/crio-b6cbcba37c8f046d04e52075fef3fe93ae52952b5d3afed515e96a437f141439 WatchSource:0}: Error finding container b6cbcba37c8f046d04e52075fef3fe93ae52952b5d3afed515e96a437f141439: Status 404 returned error can't find the container with id b6cbcba37c8f046d04e52075fef3fe93ae52952b5d3afed515e96a437f141439
Sep 18 22:46:23.196604 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:23.196566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" event={"ID":"43e91c78-e5e9-4503-aa5c-3b5851571269","Type":"ContainerStarted","Data":"fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5"}
Sep 18 22:46:23.196987 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:23.196625    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" event={"ID":"43e91c78-e5e9-4503-aa5c-3b5851571269","Type":"ContainerStarted","Data":"b6cbcba37c8f046d04e52075fef3fe93ae52952b5d3afed515e96a437f141439"}
Sep 18 22:46:23.196987 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:23.196646    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:46:23.216796 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:23.216745    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" podStartSLOduration=1.216729517 podStartE2EDuration="1.216729517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:46:23.216442741 +0000 UTC m=+3362.073098394" watchObservedRunningTime="2026-09-18 22:46:23.216729517 +0000 UTC m=+3362.073385168"
Sep 18 22:46:23.614034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:23.613997    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:46:23.614253 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:46:23.614214    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:46:26.613342 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:26.613308    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:46:26.613736 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:46:26.613535    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:46:37.613461 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:37.613431    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:46:37.613945 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:46:37.613669    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:46:39.614098 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:39.614060    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:46:39.614508 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:46:39.614314    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:46:43.320042 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:43.320010    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:46:48.613551 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:48.613524    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:46:48.613931 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:46:48.613727    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:46:53.613282 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:53.613235    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:46:53.613746 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:46:53.613432    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:46:54.202827 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:46:54.202793    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:47:03.613456 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:03.613423    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:47:03.613808 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:03.613656    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:47:08.613888 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:08.613861    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:47:08.614313 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:08.614046    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:47:11.213824 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.213779    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"]
Sep 18 22:47:11.218041 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.218020    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.220557 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.220531    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 18 22:47:11.226639 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.226618    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"]
Sep 18 22:47:11.244725 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.244699    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:47:11.286666 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.286644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.286761 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.286694    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.286761 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.286712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.286840 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.286792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.286840 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.286821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk4gz\" (UniqueName: \"kubernetes.io/projected/3c211058-9ed6-42bf-8a8c-2db5e3d80631-kube-api-access-nk4gz\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.286907 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.286854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.388096 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.388062    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.388224 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.388104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk4gz\" (UniqueName: \"kubernetes.io/projected/3c211058-9ed6-42bf-8a8c-2db5e3d80631-kube-api-access-nk4gz\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.388224 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.388140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.388224 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.388168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.388422 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.388243    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.388422 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.388292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.388657 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.388633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.388741 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.388689    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.388811 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.388745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.390619 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.390599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.390619 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.390613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.396475 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.396456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk4gz\" (UniqueName: \"kubernetes.io/projected/3c211058-9ed6-42bf-8a8c-2db5e3d80631-kube-api-access-nk4gz\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.529619 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.529584    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:11.800386 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:11.800361    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"]
Sep 18 22:47:11.802727 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:47:11.802695    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c211058_9ed6_42bf_8a8c_2db5e3d80631.slice/crio-f3cb5479bcf03876cd22cf048dd8f5953a1a7ecfd574697fe9d058e4b006f911 WatchSource:0}: Error finding container f3cb5479bcf03876cd22cf048dd8f5953a1a7ecfd574697fe9d058e4b006f911: Status 404 returned error can't find the container with id f3cb5479bcf03876cd22cf048dd8f5953a1a7ecfd574697fe9d058e4b006f911
Sep 18 22:47:12.413025 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:12.412987    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:47:12.433381 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:12.433350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh" event={"ID":"3c211058-9ed6-42bf-8a8c-2db5e3d80631","Type":"ContainerStarted","Data":"8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7"}
Sep 18 22:47:12.433507 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:12.433405    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh" event={"ID":"3c211058-9ed6-42bf-8a8c-2db5e3d80631","Type":"ContainerStarted","Data":"f3cb5479bcf03876cd22cf048dd8f5953a1a7ecfd574697fe9d058e4b006f911"}
Sep 18 22:47:12.451996 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:12.451950    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh" podStartSLOduration=1.451935864 podStartE2EDuration="1.451935864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:47:12.450478827 +0000 UTC m=+3411.307134479" watchObservedRunningTime="2026-09-18 22:47:12.451935864 +0000 UTC m=+3411.308591515"
Sep 18 22:47:12.530353 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:12.530327    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:16.614315 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:16.614228    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:47:16.614660 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:16.614450    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:47:19.613734 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:19.613704    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:47:19.614099 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:19.613912    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:47:21.530096 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:21.530058    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:21.542835 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:21.542811    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:22.493982 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:22.493948    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:30.613768 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:30.613742    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:47:30.614132 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:30.613955    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:47:32.613643 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:32.613609    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:47:32.614077 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:32.613788    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:47:38.762465 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:38.762426    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:47:43.620173 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:43.620142    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:47:43.620696 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:43.620366    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:47:44.614600 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:44.614573    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:47:44.614765 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:44.614718    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:47:44.766511 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:44.766477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:47:44.831098 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:44.831062    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:47:48.139300 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:48.139246    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:47:48.237255 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:48.237220    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:47:53.735815 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:53.735781    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:47:53.737905 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:53.737878    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"]
Sep 18 22:47:53.747223 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:53.747185    2564 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:47:53.747223 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:47:53.747223 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:47:53.747223 ip-10-0-130-79 kubenswrapper[2564]:  > containerID="8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7" cmd=["/bin/sleep","15"]
Sep 18 22:47:53.747526 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:53.747245    2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh" message="" err=<
Sep 18 22:47:53.747526 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:47:53.747526 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:47:53.747526 ip-10-0-130-79 kubenswrapper[2564]:  >
Sep 18 22:47:53.747526 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:53.747279    2564 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:47:53.747526 ip-10-0-130-79 kubenswrapper[2564]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:47:53.747526 ip-10-0-130-79 kubenswrapper[2564]:         , stdout: , stderr: , exit code -1
Sep 18 22:47:53.747526 ip-10-0-130-79 kubenswrapper[2564]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh" podUID="3c211058-9ed6-42bf-8a8c-2db5e3d80631" containerName="main" containerID="cri-o://8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7"
Sep 18 22:47:53.747526 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:53.747338    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh" podUID="3c211058-9ed6-42bf-8a8c-2db5e3d80631" containerName="main" containerID="cri-o://8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7" gracePeriod=60
Sep 18 22:47:53.803953 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:53.803912    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:47:54.089066 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.089043    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:54.180423 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180393    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-home\") pod \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") "
Sep 18 22:47:54.180582 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180430    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-model-cache\") pod \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") "
Sep 18 22:47:54.180582 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180520    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tmp-dir\") pod \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") "
Sep 18 22:47:54.180582 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180561    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tls-certs\") pod \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") "
Sep 18 22:47:54.180746 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180595    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-dshm\") pod \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") "
Sep 18 22:47:54.180746 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180631    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk4gz\" (UniqueName: \"kubernetes.io/projected/3c211058-9ed6-42bf-8a8c-2db5e3d80631-kube-api-access-nk4gz\") pod \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\" (UID: \"3c211058-9ed6-42bf-8a8c-2db5e3d80631\") "
Sep 18 22:47:54.180746 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180689    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-home" (OuterVolumeSpecName: "home") pod "3c211058-9ed6-42bf-8a8c-2db5e3d80631" (UID: "3c211058-9ed6-42bf-8a8c-2db5e3d80631"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:47:54.180746 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180703    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c211058-9ed6-42bf-8a8c-2db5e3d80631" (UID: "3c211058-9ed6-42bf-8a8c-2db5e3d80631"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:47:54.180925 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180857    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c211058-9ed6-42bf-8a8c-2db5e3d80631" (UID: "3c211058-9ed6-42bf-8a8c-2db5e3d80631"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:47:54.180974 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180949    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:47:54.180974 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180965    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:47:54.181040 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.180977    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:47:54.182840 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.182817    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c211058-9ed6-42bf-8a8c-2db5e3d80631" (UID: "3c211058-9ed6-42bf-8a8c-2db5e3d80631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:47:54.182951 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.182858    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c211058-9ed6-42bf-8a8c-2db5e3d80631-kube-api-access-nk4gz" (OuterVolumeSpecName: "kube-api-access-nk4gz") pod "3c211058-9ed6-42bf-8a8c-2db5e3d80631" (UID: "3c211058-9ed6-42bf-8a8c-2db5e3d80631"). InnerVolumeSpecName "kube-api-access-nk4gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:47:54.182951 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.182908    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-dshm" (OuterVolumeSpecName: "dshm") pod "3c211058-9ed6-42bf-8a8c-2db5e3d80631" (UID: "3c211058-9ed6-42bf-8a8c-2db5e3d80631"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:47:54.282335 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.282313    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c211058-9ed6-42bf-8a8c-2db5e3d80631-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:47:54.282335 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.282334    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c211058-9ed6-42bf-8a8c-2db5e3d80631-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:47:54.282531 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.282344    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk4gz\" (UniqueName: \"kubernetes.io/projected/3c211058-9ed6-42bf-8a8c-2db5e3d80631-kube-api-access-nk4gz\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:47:54.613323 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.613238    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:47:54.613465 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:54.613439    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:47:54.643021 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.642997    2564 generic.go:356] "Generic (PLEG): container finished" podID="3c211058-9ed6-42bf-8a8c-2db5e3d80631" containerID="8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7" exitCode=0
Sep 18 22:47:54.643142 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.643042    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"
Sep 18 22:47:54.643142 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.643058    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh" event={"ID":"3c211058-9ed6-42bf-8a8c-2db5e3d80631","Type":"ContainerDied","Data":"8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7"}
Sep 18 22:47:54.643142 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.643094    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh" event={"ID":"3c211058-9ed6-42bf-8a8c-2db5e3d80631","Type":"ContainerDied","Data":"f3cb5479bcf03876cd22cf048dd8f5953a1a7ecfd574697fe9d058e4b006f911"}
Sep 18 22:47:54.643142 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.643113    2564 scope.go:117] "RemoveContainer" containerID="8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7"
Sep 18 22:47:54.652686 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.652670    2564 scope.go:117] "RemoveContainer" containerID="8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7"
Sep 18 22:47:54.652936 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:54.652916    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7\": container with ID starting with 8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7 not found: ID does not exist" containerID="8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7"
Sep 18 22:47:54.652983 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.652943    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7"} err="failed to get container status \"8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7\": rpc error: code = NotFound desc = could not find container \"8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7\": container with ID starting with 8c263ac17eb0dd2e06aa3208f854447e92d903f118ebea6f168c5c2ee28b8cf7 not found: ID does not exist"
Sep 18 22:47:54.665368 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.665333    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"]
Sep 18 22:47:54.671706 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:54.671684    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995n22sh"]
Sep 18 22:47:55.619287 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:55.619245    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c211058-9ed6-42bf-8a8c-2db5e3d80631" path="/var/lib/kubelet/pods/3c211058-9ed6-42bf-8a8c-2db5e3d80631/volumes"
Sep 18 22:47:56.613616 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:47:56.613587    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:47:56.613798 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:47:56.613760    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:48:05.614167 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:05.614140    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:48:05.614585 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:05.614343    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:48:10.614111 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:10.614081    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:48:10.614520 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:10.614242    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:48:16.613786 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:16.613757    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:48:16.614158 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:16.613905    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:48:24.613341 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:24.613315    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:48:24.613690 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:24.613484    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:48:30.613310 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:30.613259    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:48:30.613703 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:30.613480    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:48:36.491235 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:36.491193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:48:36.585602 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:36.585572    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:48:37.614201 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:37.614174    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:48:37.614598 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:37.614374    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:48:41.065596 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.065561    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"]
Sep 18 22:48:41.066198 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.066178    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c211058-9ed6-42bf-8a8c-2db5e3d80631" containerName="main"
Sep 18 22:48:41.066243 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.066203    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c211058-9ed6-42bf-8a8c-2db5e3d80631" containerName="main"
Sep 18 22:48:41.066348 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.066337    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c211058-9ed6-42bf-8a8c-2db5e3d80631" containerName="main"
Sep 18 22:48:41.069964 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.069944    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.072676 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.072650    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 18 22:48:41.072789 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.072677    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:48:41.072789 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.072676    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 22:48:41.072789 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.072726    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 22:48:41.073169 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.073151    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 22:48:41.073284 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.073159    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-9dgjn\""
Sep 18 22:48:41.080641 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.080621    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"]
Sep 18 22:48:41.103192 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.103152    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:48:41.193663 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.193641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s982b\" (UniqueName: \"kubernetes.io/projected/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kube-api-access-s982b\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.193776 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.193674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.193776 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.193754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.193879 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.193798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.193879 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.193816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.193879 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.193844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.194004 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.193937    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.194004 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.193991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.294930 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.294902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.294954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s982b\" (UniqueName: \"kubernetes.io/projected/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kube-api-access-s982b\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.294975    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295077 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295455 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295213    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295455 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295455 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295455 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295421    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295674 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295649    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295773 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295753    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.295821 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.295799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.297445 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.297419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.297709 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.297690    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.303125 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.303103    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s982b\" (UniqueName: \"kubernetes.io/projected/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kube-api-access-s982b\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.310776 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.310750    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:48:41.381356 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.381298    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:41.557103 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.557070    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:48:41.615603 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.615581    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:48:41.615746 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:41.615729    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:48:41.676490 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.676470    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"]
Sep 18 22:48:41.678210 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:48:41.678184    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c85a46c_77a5_4ed5_b6ee_ee97f1befbef.slice/crio-cbc8f8e48dccdd529ac27ad331f9a7148fe2a0de9ae93a14cd0d2acd9aa67205 WatchSource:0}: Error finding container cbc8f8e48dccdd529ac27ad331f9a7148fe2a0de9ae93a14cd0d2acd9aa67205: Status 404 returned error can't find the container with id cbc8f8e48dccdd529ac27ad331f9a7148fe2a0de9ae93a14cd0d2acd9aa67205
Sep 18 22:48:41.868524 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.868484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" event={"ID":"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef","Type":"ContainerStarted","Data":"7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079"}
Sep 18 22:48:41.868801 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:41.868546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" event={"ID":"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef","Type":"ContainerStarted","Data":"cbc8f8e48dccdd529ac27ad331f9a7148fe2a0de9ae93a14cd0d2acd9aa67205"}
Sep 18 22:48:44.883196 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:44.883158    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerID="7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079" exitCode=0
Sep 18 22:48:44.883637 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:44.883234    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" event={"ID":"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef","Type":"ContainerDied","Data":"7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079"}
Sep 18 22:48:45.890759 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:45.890721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" event={"ID":"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef","Type":"ContainerStarted","Data":"1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590"}
Sep 18 22:48:45.914764 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:45.914708    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podStartSLOduration=2.788516778 podStartE2EDuration="4.914689135s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.126172357s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:48:45.912500762 +0000 UTC m=+3504.769156414" watchObservedRunningTime="2026-09-18 22:48:45.914689135 +0000 UTC m=+3504.771344788"
Sep 18 22:48:48.613604 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:48.613567    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:48:48.614055 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:48.613764    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:48:51.381689 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:51.381649    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:51.382191 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:51.381725    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:48:51.383091 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:51.383060    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:48:51.448039 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:51.447992    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:48:52.614180 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:52.614138    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:48:52.614687 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:52.614339    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:48:53.700308 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:53.700250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:48:59.613813 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:48:59.613781    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:48:59.614187 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:48:59.614021    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:49:00.307354 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:00.307320    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"]
Sep 18 22:49:00.322841 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:00.322805    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:00.403656 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:00.403624    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:01.382344 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:01.382295    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:49:02.937583 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:02.937546    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:04.202776 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:04.202738    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:49:04.597682 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:04.597649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:04.614084 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:04.614060    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:49:04.614304 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:04.614250    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:49:10.326260 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:10.326220    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:11.382687 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:11.382646    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:49:11.616344 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:11.616318    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:49:11.616534 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:11.616513    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:49:14.202948 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:14.202906    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:49:15.327124 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:15.327081    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerName="main" containerID="cri-o://fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5" gracePeriod=45
Sep 18 22:49:17.614562 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:17.614534    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:49:17.614929 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:17.614743    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:49:21.381730 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:21.381684    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:49:24.203518 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:24.203475    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:49:24.203997 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:24.203577    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:49:26.614454 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:26.614416    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:49:27.112895 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:27.112863    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/13.log"
Sep 18 22:49:27.113421 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:27.113403    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/12.log"
Sep 18 22:49:27.113486 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:27.113450    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9" exitCode=1
Sep 18 22:49:27.113548 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:27.113527    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"}
Sep 18 22:49:27.113599 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:27.113589    2564 scope.go:117] "RemoveContainer" containerID="81d22d4cdc6d2d3b9ae1249776237138bb0bfb271dea5af262b34f61f80d7277"
Sep 18 22:49:27.113957 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:27.113938    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:49:27.114280 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:27.114243    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:49:28.121366 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:28.121335    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/13.log"
Sep 18 22:49:28.614474 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:28.614449    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:49:28.614635 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:28.614619    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:49:29.208770 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:29.208724    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:31.381793 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:31.381739    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:49:34.179500 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:34.179470    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:49:34.179895 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:34.179525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:49:34.179895 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:34.179881    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:49:34.180120 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:34.180100    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:49:34.202742 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:34.202705    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:49:34.254218 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:34.254178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:34.353872 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:34.353836    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:40.613511 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:40.613480    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:49:40.613889 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:40.613679    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:49:41.381795 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:41.381750    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:49:44.203252 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:44.203207    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:49:45.584901 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:45.584826    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:49:45.615494 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:45.615462    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls5sz\" (UniqueName: \"kubernetes.io/projected/43e91c78-e5e9-4503-aa5c-3b5851571269-kube-api-access-ls5sz\") pod \"43e91c78-e5e9-4503-aa5c-3b5851571269\" (UID: \"43e91c78-e5e9-4503-aa5c-3b5851571269\") "
Sep 18 22:49:45.615696 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:45.615550    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e91c78-e5e9-4503-aa5c-3b5851571269-tls-certs\") pod \"43e91c78-e5e9-4503-aa5c-3b5851571269\" (UID: \"43e91c78-e5e9-4503-aa5c-3b5851571269\") "
Sep 18 22:49:45.617764 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:45.617715    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e91c78-e5e9-4503-aa5c-3b5851571269-kube-api-access-ls5sz" (OuterVolumeSpecName: "kube-api-access-ls5sz") pod "43e91c78-e5e9-4503-aa5c-3b5851571269" (UID: "43e91c78-e5e9-4503-aa5c-3b5851571269"). InnerVolumeSpecName "kube-api-access-ls5sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:49:45.617898 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:45.617844    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e91c78-e5e9-4503-aa5c-3b5851571269-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43e91c78-e5e9-4503-aa5c-3b5851571269" (UID: "43e91c78-e5e9-4503-aa5c-3b5851571269"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:49:45.716859 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:45.716817    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls5sz\" (UniqueName: \"kubernetes.io/projected/43e91c78-e5e9-4503-aa5c-3b5851571269-kube-api-access-ls5sz\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:49:45.716859 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:45.716851    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e91c78-e5e9-4503-aa5c-3b5851571269-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:49:46.214798 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:46.214766    2564 generic.go:356] "Generic (PLEG): container finished" podID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerID="fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5" exitCode=0
Sep 18 22:49:46.214980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:46.214817    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"
Sep 18 22:49:46.214980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:46.214851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" event={"ID":"43e91c78-e5e9-4503-aa5c-3b5851571269","Type":"ContainerDied","Data":"fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5"}
Sep 18 22:49:46.214980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:46.214920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f" event={"ID":"43e91c78-e5e9-4503-aa5c-3b5851571269","Type":"ContainerDied","Data":"b6cbcba37c8f046d04e52075fef3fe93ae52952b5d3afed515e96a437f141439"}
Sep 18 22:49:46.214980 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:46.214948    2564 scope.go:117] "RemoveContainer" containerID="fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5"
Sep 18 22:49:46.223809 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:46.223782    2564 scope.go:117] "RemoveContainer" containerID="fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5"
Sep 18 22:49:46.224111 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:46.224089    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5\": container with ID starting with fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5 not found: ID does not exist" containerID="fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5"
Sep 18 22:49:46.224181 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:46.224121    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5"} err="failed to get container status \"fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5\": rpc error: code = NotFound desc = could not find container \"fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5\": container with ID starting with fa240dfcc5743d99bb3143c534e6a27db9998550c31dcff30c924c505e52b8d5 not found: ID does not exist"
Sep 18 22:49:46.233319 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:46.233285    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"]
Sep 18 22:49:46.235763 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:46.235742    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6m79f"]
Sep 18 22:49:47.614429 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:47.614401    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:49:47.614805 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:47.614597    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:49:47.622598 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:47.622569    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" path="/var/lib/kubelet/pods/43e91c78-e5e9-4503-aa5c-3b5851571269/volumes"
Sep 18 22:49:49.280314 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:49.280257    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:49.374735 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:49.374690    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:49:51.382545 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:51.382507    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:49:52.614388 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:52.614355    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:49:52.614776 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:49:52.614624    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:49:55.802094 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:49:55.802055    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:50:01.382567 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:01.382514    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:50:01.386437 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:01.386412    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:50:01.487316 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:01.487262    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:50:02.613822 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:02.613794    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:50:02.614206 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:02.613983    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:50:03.613399 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:03.613369    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:50:03.613576 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:03.613556    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:50:04.550874 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:04.550830    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:50:04.744230 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:04.744194    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:50:11.382436 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:11.382394    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:50:13.613328 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:13.613300    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:50:13.613767 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:13.613485    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:50:17.619800 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:17.619772    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:50:17.620281 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:17.620053    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:50:21.382152 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:21.382115    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:50:21.897950 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:21.897917    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/9.log"
Sep 18 22:50:21.898728 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:21.898705    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/13.log"
Sep 18 22:50:21.904094 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:21.904077    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:50:21.910804 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:21.910780    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/9.log"
Sep 18 22:50:21.911636 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:21.911616    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/13.log"
Sep 18 22:50:21.916837 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:21.916815    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:50:22.377960 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:22.377916    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:50:28.613953 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:28.613925    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:50:28.614388 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:28.614113    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:50:31.381936 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:31.381889    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 22:50:32.613767 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:32.613737    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:50:33.461401 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:33.461374    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/10.log"
Sep 18 22:50:33.461880 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:33.461861    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/9.log"
Sep 18 22:50:33.461967 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:33.461898    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737" exitCode=1
Sep 18 22:50:33.461967 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:33.461925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"}
Sep 18 22:50:33.462040 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:33.461967    2564 scope.go:117] "RemoveContainer" containerID="68fdb7e9684067e1e6a5b8735b4a51b7fdd7cbfa6498074fff868b15404474b4"
Sep 18 22:50:33.462351 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:33.462329    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:50:33.462577 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:33.462555    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:50:34.468442 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:34.468413    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/10.log"
Sep 18 22:50:39.613950 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:39.613916    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:50:39.614362 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:39.614095    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:50:41.392575 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:41.392543    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:50:41.405233 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:41.405203    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:50:41.727456 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:41.727380    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:50:41.727456 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:41.727430    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:50:41.727672 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:41.727660    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:50:41.727846 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:41.727829    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:50:43.151726 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.151684    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:50:43.324656 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.324624    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"]
Sep 18 22:50:43.325006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.324994    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerName="main"
Sep 18 22:50:43.325057 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.325007    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerName="main"
Sep 18 22:50:43.325092 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.325074    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e91c78-e5e9-4503-aa5c-3b5851571269" containerName="main"
Sep 18 22:50:43.328090 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.328074    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:50:43.331231 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.331198    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 18 22:50:43.331231 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.331220    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 22:50:43.331447 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.331235    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 22:50:43.331447 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.331370    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-rbr4p\""
Sep 18 22:50:43.331610 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.331592    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-954w7\""
Sep 18 22:50:43.339342 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.339321    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"]
Sep 18 22:50:43.420653 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.420542    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:50:43.503127 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.503091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649f6ebf-980a-406e-b474-2da6e838b97f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp\" (UID: \"649f6ebf-980a-406e-b474-2da6e838b97f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:50:43.503304 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.503147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kqv\" (UniqueName: \"kubernetes.io/projected/649f6ebf-980a-406e-b474-2da6e838b97f-kube-api-access-v8kqv\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp\" (UID: \"649f6ebf-980a-406e-b474-2da6e838b97f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:50:43.603678 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.603651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649f6ebf-980a-406e-b474-2da6e838b97f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp\" (UID: \"649f6ebf-980a-406e-b474-2da6e838b97f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:50:43.603821 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.603703    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kqv\" (UniqueName: \"kubernetes.io/projected/649f6ebf-980a-406e-b474-2da6e838b97f-kube-api-access-v8kqv\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp\" (UID: \"649f6ebf-980a-406e-b474-2da6e838b97f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:50:43.606095 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.606068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649f6ebf-980a-406e-b474-2da6e838b97f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp\" (UID: \"649f6ebf-980a-406e-b474-2da6e838b97f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:50:43.612189 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.612163    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kqv\" (UniqueName: \"kubernetes.io/projected/649f6ebf-980a-406e-b474-2da6e838b97f-kube-api-access-v8kqv\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp\" (UID: \"649f6ebf-980a-406e-b474-2da6e838b97f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:50:43.639396 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.639375    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:50:43.910792 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:43.910766    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"]
Sep 18 22:50:43.912861 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:50:43.912832    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649f6ebf_980a_406e_b474_2da6e838b97f.slice/crio-93189a52ea2cfb221a056659220ccb14017d58ae3a9be4f7a7212f72181587ae WatchSource:0}: Error finding container 93189a52ea2cfb221a056659220ccb14017d58ae3a9be4f7a7212f72181587ae: Status 404 returned error can't find the container with id 93189a52ea2cfb221a056659220ccb14017d58ae3a9be4f7a7212f72181587ae
Sep 18 22:50:44.518966 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:44.518926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" event={"ID":"649f6ebf-980a-406e-b474-2da6e838b97f","Type":"ContainerStarted","Data":"988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6"}
Sep 18 22:50:44.519380 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:44.518986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" event={"ID":"649f6ebf-980a-406e-b474-2da6e838b97f","Type":"ContainerStarted","Data":"93189a52ea2cfb221a056659220ccb14017d58ae3a9be4f7a7212f72181587ae"}
Sep 18 22:50:44.519380 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:44.519135    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:50:44.537056 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:44.537005    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" podStartSLOduration=1.536990439 podStartE2EDuration="1.536990439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:50:44.534999234 +0000 UTC m=+3623.391654887" watchObservedRunningTime="2026-09-18 22:50:44.536990439 +0000 UTC m=+3623.393646090"
Sep 18 22:50:54.613768 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:54.613738    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:50:54.614149 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:54.613904    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:50:55.613715 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:50:55.613690    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:50:55.613912 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:50:55.613891    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:51:05.614040 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:05.614013    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:51:05.614481 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:05.614203    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:51:09.714834 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:09.714799    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:51:10.613425 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:10.613400    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:51:10.613620 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:10.613602    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:51:11.730991 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:11.730958    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:51:11.739932 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:11.739906    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"]
Sep 18 22:51:11.801027 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:11.800998    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:51:15.525515 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:15.525488    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:51:20.614149 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:20.614118    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:51:20.614582 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:20.614336    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:51:21.462421 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:21.462367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:51:21.475353 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:21.475326    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"]
Sep 18 22:51:21.553887 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:21.553859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:51:25.525550 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:25.525508    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:51:25.619358 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:25.619330    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:51:25.619555 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:25.619533    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:51:26.762843 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:26.762800    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main" containerID="cri-o://1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590" gracePeriod=45
Sep 18 22:51:27.502430 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.502405    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:51:27.597729 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.597657    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kserve-provision-location\") pod \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") "
Sep 18 22:51:27.597729 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.597700    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tmp-dir\") pod \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") "
Sep 18 22:51:27.597729 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.597721    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-cabundle-cert\") pod \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") "
Sep 18 22:51:27.597985 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.597766    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tls-certs\") pod \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") "
Sep 18 22:51:27.597985 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.597784    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s982b\" (UniqueName: \"kubernetes.io/projected/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kube-api-access-s982b\") pod \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") "
Sep 18 22:51:27.597985 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.597820    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-dshm\") pod \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") "
Sep 18 22:51:27.597985 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.597876    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-model-cache\") pod \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") "
Sep 18 22:51:27.597985 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.597923    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-home\") pod \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\" (UID: \"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef\") "
Sep 18 22:51:27.598242 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.598159    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" (UID: "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:51:27.598338 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.598310    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" (UID: "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.598861 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.598832    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-home" (OuterVolumeSpecName: "home") pod "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" (UID: "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.600224 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.600198    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-dshm" (OuterVolumeSpecName: "dshm") pod "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" (UID: "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.600430 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.600382    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" (UID: "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:51:27.600547 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.600445    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kube-api-access-s982b" (OuterVolumeSpecName: "kube-api-access-s982b") pod "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" (UID: "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef"). InnerVolumeSpecName "kube-api-access-s982b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:51:27.610500 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.610478    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" (UID: "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.622125 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.622104    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" (UID: "7c85a46c-77a5-4ed5-b6ee-ee97f1befbef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.699155 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.699135    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.699155 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.699155    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s982b\" (UniqueName: \"kubernetes.io/projected/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kube-api-access-s982b\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.699303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.699165    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.699303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.699174    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.699303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.699186    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.699303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.699198    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.699303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.699206    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.699303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.699213    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.729398 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.729370    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerID="1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590" exitCode=0
Sep 18 22:51:27.729509 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.729429    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"
Sep 18 22:51:27.729509 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.729454    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" event={"ID":"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef","Type":"ContainerDied","Data":"1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590"}
Sep 18 22:51:27.729621 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.729511    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n" event={"ID":"7c85a46c-77a5-4ed5-b6ee-ee97f1befbef","Type":"ContainerDied","Data":"cbc8f8e48dccdd529ac27ad331f9a7148fe2a0de9ae93a14cd0d2acd9aa67205"}
Sep 18 22:51:27.729621 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.729534    2564 scope.go:117] "RemoveContainer" containerID="1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590"
Sep 18 22:51:27.738202 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.738191    2564 scope.go:117] "RemoveContainer" containerID="7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079"
Sep 18 22:51:27.746218 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.746198    2564 scope.go:117] "RemoveContainer" containerID="1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590"
Sep 18 22:51:27.746525 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:27.746509    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590\": container with ID starting with 1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590 not found: ID does not exist" containerID="1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590"
Sep 18 22:51:27.746577 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.746534    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590"} err="failed to get container status \"1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590\": rpc error: code = NotFound desc = could not find container \"1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590\": container with ID starting with 1407ae73f4f95c015e7a06facbee56ae85e40d6962df7d99be4b3a5701d94590 not found: ID does not exist"
Sep 18 22:51:27.746577 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.746548    2564 scope.go:117] "RemoveContainer" containerID="7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079"
Sep 18 22:51:27.746780 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:27.746764    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079\": container with ID starting with 7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079 not found: ID does not exist" containerID="7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079"
Sep 18 22:51:27.746822 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.746785    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079"} err="failed to get container status \"7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079\": rpc error: code = NotFound desc = could not find container \"7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079\": container with ID starting with 7651a0924a8c503e1104c6b332c0f1866c1c4950a954aac1029ab7c3e1182079 not found: ID does not exist"
Sep 18 22:51:27.753560 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.753540    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"]
Sep 18 22:51:27.757827 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:27.757805    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54d882n"]
Sep 18 22:51:29.619579 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:29.619549    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" path="/var/lib/kubelet/pods/7c85a46c-77a5-4ed5-b6ee-ee97f1befbef/volumes"
Sep 18 22:51:34.613249 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:34.613213    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:51:34.613812 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:34.613522    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:51:35.525409 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:35.525375    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:51:36.494639 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:36.494597    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" containerName="main" containerID="cri-o://988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6" gracePeriod=45
Sep 18 22:51:39.614359 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:39.614330    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:51:39.614742 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:39.614521    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:51:45.525614 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:45.525528    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:51:45.525959 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:45.525633    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:51:49.613372 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:49.613347    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:51:49.613746 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:49.613532    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:51:50.613457 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:50.613428    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:51:50.613810 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:51:50.613634    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:51:55.526061 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:55.526020    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:51:58.915512 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:58.915476    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:51:59.011514 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:51:59.011470    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:00.613932 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:00.613901    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:52:00.614396 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:00.614162    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:52:03.613490 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:03.613459    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:52:03.613870 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:03.613682    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:52:05.305322 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:05.305286    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:05.525330 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:05.525294    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:52:06.736465 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.736441    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:52:06.830877 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.830833    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8kqv\" (UniqueName: \"kubernetes.io/projected/649f6ebf-980a-406e-b474-2da6e838b97f-kube-api-access-v8kqv\") pod \"649f6ebf-980a-406e-b474-2da6e838b97f\" (UID: \"649f6ebf-980a-406e-b474-2da6e838b97f\") "
Sep 18 22:52:06.831017 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.830946    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649f6ebf-980a-406e-b474-2da6e838b97f-tls-certs\") pod \"649f6ebf-980a-406e-b474-2da6e838b97f\" (UID: \"649f6ebf-980a-406e-b474-2da6e838b97f\") "
Sep 18 22:52:06.832829 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.832799    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649f6ebf-980a-406e-b474-2da6e838b97f-kube-api-access-v8kqv" (OuterVolumeSpecName: "kube-api-access-v8kqv") pod "649f6ebf-980a-406e-b474-2da6e838b97f" (UID: "649f6ebf-980a-406e-b474-2da6e838b97f"). InnerVolumeSpecName "kube-api-access-v8kqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:52:06.832945 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.832871    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649f6ebf-980a-406e-b474-2da6e838b97f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "649f6ebf-980a-406e-b474-2da6e838b97f" (UID: "649f6ebf-980a-406e-b474-2da6e838b97f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:52:06.912702 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.912672    2564 generic.go:356] "Generic (PLEG): container finished" podID="649f6ebf-980a-406e-b474-2da6e838b97f" containerID="988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6" exitCode=0
Sep 18 22:52:06.912807 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.912716    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"
Sep 18 22:52:06.912807 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.912755    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" event={"ID":"649f6ebf-980a-406e-b474-2da6e838b97f","Type":"ContainerDied","Data":"988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6"}
Sep 18 22:52:06.912883 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.912806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp" event={"ID":"649f6ebf-980a-406e-b474-2da6e838b97f","Type":"ContainerDied","Data":"93189a52ea2cfb221a056659220ccb14017d58ae3a9be4f7a7212f72181587ae"}
Sep 18 22:52:06.912883 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.912830    2564 scope.go:117] "RemoveContainer" containerID="988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6"
Sep 18 22:52:06.921153 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.921139    2564 scope.go:117] "RemoveContainer" containerID="988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6"
Sep 18 22:52:06.921431 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:06.921411    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6\": container with ID starting with 988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6 not found: ID does not exist" containerID="988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6"
Sep 18 22:52:06.921508 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.921438    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6"} err="failed to get container status \"988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6\": rpc error: code = NotFound desc = could not find container \"988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6\": container with ID starting with 988e9a57ec21bcb169ff5b84286ccab1ae5826fa05628bdad1471303a67210a6 not found: ID does not exist"
Sep 18 22:52:06.932318 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.932296    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649f6ebf-980a-406e-b474-2da6e838b97f-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:52:06.932402 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.932320    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8kqv\" (UniqueName: \"kubernetes.io/projected/649f6ebf-980a-406e-b474-2da6e838b97f-kube-api-access-v8kqv\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:52:06.938476 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.938457    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"]
Sep 18 22:52:06.952026 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:06.951998    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemhwtp"]
Sep 18 22:52:07.619284 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:07.619242    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" path="/var/lib/kubelet/pods/649f6ebf-980a-406e-b474-2da6e838b97f/volumes"
Sep 18 22:52:11.042018 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:11.041979    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:11.141198 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:11.141173    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:11.615769 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:11.615734    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:52:11.615973 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:11.615921    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:52:14.997782 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:14.997750    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"]
Sep 18 22:52:14.998312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:14.998149    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="storage-initializer"
Sep 18 22:52:14.998312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:14.998161    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="storage-initializer"
Sep 18 22:52:14.998312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:14.998179    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main"
Sep 18 22:52:14.998312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:14.998185    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main"
Sep 18 22:52:14.998312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:14.998202    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" containerName="main"
Sep 18 22:52:14.998312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:14.998207    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" containerName="main"
Sep 18 22:52:14.998312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:14.998282    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="649f6ebf-980a-406e-b474-2da6e838b97f" containerName="main"
Sep 18 22:52:14.998312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:14.998296    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c85a46c-77a5-4ed5-b6ee-ee97f1befbef" containerName="main"
Sep 18 22:52:15.001584 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.001561    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.004207 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.004182    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 22:52:15.004358 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.004233    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 22:52:15.004358 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.004261    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-bcvzt\""
Sep 18 22:52:15.004487 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.004358    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 22:52:15.004552 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.004535    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:52:15.005162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.005143    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 22:52:15.013499 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.013477    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"]
Sep 18 22:52:15.027948 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.027921    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:15.103075 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.103043    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.103075 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.103078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.103312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.103097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4h4\" (UniqueName: \"kubernetes.io/projected/52239568-6075-4753-8971-aa183a32b530-kube-api-access-ks4h4\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.103312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.103124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52239568-6075-4753-8971-aa183a32b530-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.103312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.103160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52239568-6075-4753-8971-aa183a32b530-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.103312 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.103253    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.103474 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.103354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.103474 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.103374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204105 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204068    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204308 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204308 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204388 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204388 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204483 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4h4\" (UniqueName: \"kubernetes.io/projected/52239568-6075-4753-8971-aa183a32b530-kube-api-access-ks4h4\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204483 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52239568-6075-4753-8971-aa183a32b530-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204483 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52239568-6075-4753-8971-aa183a32b530-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204640 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204530    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204640 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204755 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204716    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.204800 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.204751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.205070 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.205045    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52239568-6075-4753-8971-aa183a32b530-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.206482 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.206458    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.206877 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.206858    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52239568-6075-4753-8971-aa183a32b530-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.213556 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.213533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4h4\" (UniqueName: \"kubernetes.io/projected/52239568-6075-4753-8971-aa183a32b530-kube-api-access-ks4h4\") pod \"custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.312378 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.312340    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:15.594890 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.594861    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"]
Sep 18 22:52:15.596459 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:52:15.596430    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52239568_6075_4753_8971_aa183a32b530.slice/crio-3e4f615d8c6e1ea173ced99834fe60426d95f777131c78553a82e9798fbdfe43 WatchSource:0}: Error finding container 3e4f615d8c6e1ea173ced99834fe60426d95f777131c78553a82e9798fbdfe43: Status 404 returned error can't find the container with id 3e4f615d8c6e1ea173ced99834fe60426d95f777131c78553a82e9798fbdfe43
Sep 18 22:52:15.957812 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.957731    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" event={"ID":"52239568-6075-4753-8971-aa183a32b530","Type":"ContainerStarted","Data":"8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209"}
Sep 18 22:52:15.957812 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:15.957784    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" event={"ID":"52239568-6075-4753-8971-aa183a32b530","Type":"ContainerStarted","Data":"3e4f615d8c6e1ea173ced99834fe60426d95f777131c78553a82e9798fbdfe43"}
Sep 18 22:52:16.236991 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:16.236912    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:16.613880 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:16.613844    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:52:16.614089 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:16.614043    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:52:17.970856 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:17.970822    2564 generic.go:356] "Generic (PLEG): container finished" podID="52239568-6075-4753-8971-aa183a32b530" containerID="8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209" exitCode=0
Sep 18 22:52:17.971316 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:17.970897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" event={"ID":"52239568-6075-4753-8971-aa183a32b530","Type":"ContainerDied","Data":"8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209"}
Sep 18 22:52:18.493748 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:18.493716    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:18.596521 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:18.596490    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:18.982285 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:18.982217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" event={"ID":"52239568-6075-4753-8971-aa183a32b530","Type":"ContainerStarted","Data":"6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15"}
Sep 18 22:52:19.009510 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:19.009443    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podStartSLOduration=2.83509048 podStartE2EDuration="5.009422282s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.174331802s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:52:19.005527116 +0000 UTC m=+3717.862182769" watchObservedRunningTime="2026-09-18 22:52:19.009422282 +0000 UTC m=+3717.866077935"
Sep 18 22:52:23.613908 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:23.613877    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:52:23.614377 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:23.614053    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:52:25.313515 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:25.313474    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:25.313973 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:25.313554    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:52:25.315113 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:25.315084    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:52:27.766636 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:27.766550    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:29.620499 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:29.620470    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:52:29.620869 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:29.620747    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:52:34.749904 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:34.749871    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:34.820593 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:34.820564    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:35.313712 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:35.313668    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:52:37.872814 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:37.872780    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:37.970153 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:37.970123    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:52:38.614049 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:38.614021    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:52:38.614228 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:38.614208    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:52:40.613453 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:40.613424    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:52:40.613818 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:40.613619    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:52:45.313063 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:45.313020    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:52:51.616091 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:51.616059    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:52:51.616549 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:51.616335    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:52:54.613226 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:54.613189    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:52:54.613764 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:52:54.613509    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:52:55.313032 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:52:55.312982    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:53:04.614056 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:04.614027    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:53:04.614498 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:53:04.614236    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:53:05.313316 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:05.313247    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:53:09.614666 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:09.614635    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:53:09.615056 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:53:09.614814    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:53:15.313823 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:15.313723    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:53:15.614908 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:15.614828    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:53:15.615062 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:53:15.615042    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:53:20.614210 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:20.614164    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:53:20.614798 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:53:20.614539    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:53:21.710968 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:21.710935    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:53:21.804962 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:21.804931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:53:25.313319 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:25.313253    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:53:28.614222 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:28.614185    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:53:28.614693 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:53:28.614414    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:53:28.693060 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:28.693024    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:53:30.562933 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:30.562902    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:53:30.811564 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:30.811526    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:53:33.614038 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:33.614003    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:53:33.614606 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:53:33.614262    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:53:35.055447 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:35.055403    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:53:35.156966 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:35.156930    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:53:35.313608 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:35.313494    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:53:38.842512 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:38.842476    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:53:43.613793 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:43.613762    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:53:43.614186 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:53:43.613950    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:53:45.313622 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:45.313576    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:53:48.613587 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:48.613549    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:53:48.614029 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:53:48.613759    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:53:55.313031 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:55.312979    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:53:58.614351 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:53:58.614315    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:53:58.614856 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:53:58.614503    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:54:01.616498 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:01.616453    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:54:01.617190 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:01.616726    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:54:05.313017 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:05.312968    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 18 22:54:09.614003 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:09.613973    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:54:09.614424 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:09.614157    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:54:12.613658 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:12.613628    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:54:12.614022 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:12.613812    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:54:15.322633 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:15.322606    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:54:15.330648 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:15.330622    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:54:21.616018 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:21.615987    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:54:21.616435 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:21.616215    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:54:22.856174 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:22.856144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:54:23.613758 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:23.613732    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:54:23.613942 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:23.613922    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:54:24.535415 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:24.535383    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"]
Sep 18 22:54:24.538604 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:24.538574    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:54:24.604214 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:24.604185    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:54:34.613589 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:34.613561    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:54:34.614102 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:34.613769    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:54:35.614193 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:35.614157    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:54:36.675116 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:36.675084    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/14.log"
Sep 18 22:54:36.675576 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:36.675559    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/13.log"
Sep 18 22:54:36.675620 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:36.675595    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50" exitCode=1
Sep 18 22:54:36.675696 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:36.675676    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"}
Sep 18 22:54:36.675748 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:36.675736    2564 scope.go:117] "RemoveContainer" containerID="628be2cb4d71445903208e7f420b2df0b09c120c1bf9518658e47dd85cb562e9"
Sep 18 22:54:36.676550 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:36.676530    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:54:36.682446 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:36.677093    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:54:37.681702 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:37.681675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/14.log"
Sep 18 22:54:39.558012 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:39.557969    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main" containerID="cri-o://6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15" gracePeriod=45
Sep 18 22:54:40.306152 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.306127    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:54:40.374884 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.374817    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-kserve-provision-location\") pod \"52239568-6075-4753-8971-aa183a32b530\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") "
Sep 18 22:54:40.374884 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.374849    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52239568-6075-4753-8971-aa183a32b530-tls-certs\") pod \"52239568-6075-4753-8971-aa183a32b530\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") "
Sep 18 22:54:40.375034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.374868    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-home\") pod \"52239568-6075-4753-8971-aa183a32b530\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") "
Sep 18 22:54:40.375034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.374966    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52239568-6075-4753-8971-aa183a32b530-cabundle-cert\") pod \"52239568-6075-4753-8971-aa183a32b530\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") "
Sep 18 22:54:40.375034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.375020    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-dshm\") pod \"52239568-6075-4753-8971-aa183a32b530\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") "
Sep 18 22:54:40.375162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.375043    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-tmp-dir\") pod \"52239568-6075-4753-8971-aa183a32b530\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") "
Sep 18 22:54:40.375162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.375089    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-model-cache\") pod \"52239568-6075-4753-8971-aa183a32b530\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") "
Sep 18 22:54:40.375162 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.375119    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks4h4\" (UniqueName: \"kubernetes.io/projected/52239568-6075-4753-8971-aa183a32b530-kube-api-access-ks4h4\") pod \"52239568-6075-4753-8971-aa183a32b530\" (UID: \"52239568-6075-4753-8971-aa183a32b530\") "
Sep 18 22:54:40.375454 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.375349    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52239568-6075-4753-8971-aa183a32b530-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52239568-6075-4753-8971-aa183a32b530" (UID: "52239568-6075-4753-8971-aa183a32b530"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:54:40.375537 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.375472    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52239568-6075-4753-8971-aa183a32b530-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.375537 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.375510    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-model-cache" (OuterVolumeSpecName: "model-cache") pod "52239568-6075-4753-8971-aa183a32b530" (UID: "52239568-6075-4753-8971-aa183a32b530"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.375695 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.375654    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-home" (OuterVolumeSpecName: "home") pod "52239568-6075-4753-8971-aa183a32b530" (UID: "52239568-6075-4753-8971-aa183a32b530"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.377106 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.377083    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52239568-6075-4753-8971-aa183a32b530-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52239568-6075-4753-8971-aa183a32b530" (UID: "52239568-6075-4753-8971-aa183a32b530"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:54:40.377550 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.377531    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-dshm" (OuterVolumeSpecName: "dshm") pod "52239568-6075-4753-8971-aa183a32b530" (UID: "52239568-6075-4753-8971-aa183a32b530"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.377630 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.377552    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52239568-6075-4753-8971-aa183a32b530-kube-api-access-ks4h4" (OuterVolumeSpecName: "kube-api-access-ks4h4") pod "52239568-6075-4753-8971-aa183a32b530" (UID: "52239568-6075-4753-8971-aa183a32b530"). InnerVolumeSpecName "kube-api-access-ks4h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:54:40.387116 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.387093    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52239568-6075-4753-8971-aa183a32b530" (UID: "52239568-6075-4753-8971-aa183a32b530"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.406543 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.406522    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52239568-6075-4753-8971-aa183a32b530" (UID: "52239568-6075-4753-8971-aa183a32b530"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.476362 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.476339    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.476362 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.476360    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4h4\" (UniqueName: \"kubernetes.io/projected/52239568-6075-4753-8971-aa183a32b530-kube-api-access-ks4h4\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.476493 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.476369    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.476493 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.476379    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52239568-6075-4753-8971-aa183a32b530-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.476493 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.476389    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.476493 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.476396    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.476493 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.476403    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52239568-6075-4753-8971-aa183a32b530-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.696654 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.696573    2564 generic.go:356] "Generic (PLEG): container finished" podID="52239568-6075-4753-8971-aa183a32b530" containerID="6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15" exitCode=0
Sep 18 22:54:40.696654 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.696619    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" event={"ID":"52239568-6075-4753-8971-aa183a32b530","Type":"ContainerDied","Data":"6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15"}
Sep 18 22:54:40.696654 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.696634    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"
Sep 18 22:54:40.697122 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.696670    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88" event={"ID":"52239568-6075-4753-8971-aa183a32b530","Type":"ContainerDied","Data":"3e4f615d8c6e1ea173ced99834fe60426d95f777131c78553a82e9798fbdfe43"}
Sep 18 22:54:40.697122 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.696701    2564 scope.go:117] "RemoveContainer" containerID="6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15"
Sep 18 22:54:40.706114 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.706094    2564 scope.go:117] "RemoveContainer" containerID="8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209"
Sep 18 22:54:40.714017 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.713996    2564 scope.go:117] "RemoveContainer" containerID="6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15"
Sep 18 22:54:40.714310 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:40.714287    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15\": container with ID starting with 6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15 not found: ID does not exist" containerID="6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15"
Sep 18 22:54:40.714368 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.714319    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15"} err="failed to get container status \"6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15\": rpc error: code = NotFound desc = could not find container \"6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15\": container with ID starting with 6821012d85aea72827a39b0fb20035ad42d8a9e6da318173888634df666b8c15 not found: ID does not exist"
Sep 18 22:54:40.714368 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.714336    2564 scope.go:117] "RemoveContainer" containerID="8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209"
Sep 18 22:54:40.714603 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:40.714583    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209\": container with ID starting with 8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209 not found: ID does not exist" containerID="8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209"
Sep 18 22:54:40.714657 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.714610    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209"} err="failed to get container status \"8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209\": rpc error: code = NotFound desc = could not find container \"8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209\": container with ID starting with 8e1caf7ba42a674ff0c814749213d41dee71ea525533db33ba61db1b2dcc5209 not found: ID does not exist"
Sep 18 22:54:40.719402 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.719380    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"]
Sep 18 22:54:40.726078 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:40.726058    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5695f75d5d-6lp88"]
Sep 18 22:54:41.618881 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:41.618849    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52239568-6075-4753-8971-aa183a32b530" path="/var/lib/kubelet/pods/52239568-6075-4753-8971-aa183a32b530/volumes"
Sep 18 22:54:44.179214 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:44.179142    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:54:44.179214 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:44.179199    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:54:44.179607 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:44.179511    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:54:44.179700 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:44.179680    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:54:44.716746 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:44.716717    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:54:44.716942 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:44.716921    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:54:45.613331 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:45.613302    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:54:45.613792 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:45.613465    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:54:57.613644 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:57.613615    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:54:57.614056 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:57.613880    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:54:59.613882 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:59.613852    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:54:59.614298 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:54:59.614048    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:54:59.824484 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:59.824451    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:54:59.879355 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:54:59.879280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:08.613786 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:08.613757    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:55:08.614260 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:55:08.613968    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:55:13.613664 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:13.613636    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:55:13.614050 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:55:13.613801    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:55:14.652345 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:14.652307    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:14.757080 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:14.757054    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:15.003953 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:15.003907    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:15.106521 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:15.106485    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:18.318303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:18.318245    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:19.751047 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:19.751016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:19.847020 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:19.846969    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:21.259027 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:21.258994    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:21.929452 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:21.929424    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/10.log"
Sep 18 22:55:21.930178 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:21.930161    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/14.log"
Sep 18 22:55:21.934895 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:21.934879    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:55:21.944514 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:21.944492    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/10.log"
Sep 18 22:55:21.945221 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:21.945202    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/14.log"
Sep 18 22:55:21.949944 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:21.949926    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 22:55:23.613334 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:23.613308    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:55:23.613730 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:55:23.613504    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:55:25.614589 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:25.614560    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:55:25.615057 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:55:25.614805    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:55:27.423794 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:27.423761    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:33.500860 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:33.500826    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:33.597473 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:33.597446    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:34.613495 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:34.613467    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:55:34.956580 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:34.956505    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/11.log"
Sep 18 22:55:34.956961 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:34.956935    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/10.log"
Sep 18 22:55:34.957075 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:34.956976    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567" exitCode=1
Sep 18 22:55:34.957075 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:34.957032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"}
Sep 18 22:55:34.957258 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:34.957091    2564 scope.go:117] "RemoveContainer" containerID="2e1ebe598ebc2f4fa45c4343fd14eccccbafd5661ff6f51b7b0d57be55086737"
Sep 18 22:55:34.957399 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:34.957380    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:55:34.957661 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:55:34.957637    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:55:35.962859 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:35.962830    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/11.log"
Sep 18 22:55:36.614200 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:36.614173    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:55:36.614398 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:55:36.614374    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:55:41.727355 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:41.727323    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:55:41.727752 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:41.727395    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 22:55:41.727752 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:41.727670    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:55:41.727840 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:55:41.727822    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:55:48.614151 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.614121    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:55:48.614551 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:55:48.614352    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:55:48.791447 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.791407    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"]
Sep 18 22:55:48.791790 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.791778    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main"
Sep 18 22:55:48.791851 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.791792    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main"
Sep 18 22:55:48.791851 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.791828    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="storage-initializer"
Sep 18 22:55:48.791851 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.791835    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="storage-initializer"
Sep 18 22:55:48.791959 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.791899    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="52239568-6075-4753-8971-aa183a32b530" containerName="main"
Sep 18 22:55:48.794881 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.794866    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.797401 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.797375    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:55:48.797546 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.797382    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 22:55:48.797546 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.797474    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-6mhsb\""
Sep 18 22:55:48.797546 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.797474    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 22:55:48.797546 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.797524    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 22:55:48.797774 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.797525    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 22:55:48.805815 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.805791    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"]
Sep 18 22:55:48.865289 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.865174    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.865289 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.865214    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/492e9ed3-fbb3-493e-a85e-030b09caf8e9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.865289 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.865236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.865520 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.865330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-home\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.865520 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.865400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzg5p\" (UniqueName: \"kubernetes.io/projected/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kube-api-access-mzg5p\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.865520 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.865438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.865520 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.865484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.865715 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.865618    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.912230 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.912180    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:48.966876 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.966842    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.966876 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.966876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/492e9ed3-fbb3-493e-a85e-030b09caf8e9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967103 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.966898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967103 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.966928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-home\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967103 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.966960    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzg5p\" (UniqueName: \"kubernetes.io/projected/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kube-api-access-mzg5p\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967103 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.966992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967103 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.967045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967391 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.967133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967502 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.967483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-home\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967561 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.967526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967599 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.967574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967683 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.967656    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.967830 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.967667    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/492e9ed3-fbb3-493e-a85e-030b09caf8e9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.969354 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.969333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.969537 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.969520    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:48.975484 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:48.975467    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzg5p\" (UniqueName: \"kubernetes.io/projected/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kube-api-access-mzg5p\") pod \"router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:49.106006 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:49.105964    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:49.387111 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:49.387083    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"]
Sep 18 22:55:49.388588 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:55:49.388563    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492e9ed3_fbb3_493e_a85e_030b09caf8e9.slice/crio-38ef41beb82b0f00576ebe36ec4ae79b895dc9253b679be1105ca5cebea15743 WatchSource:0}: Error finding container 38ef41beb82b0f00576ebe36ec4ae79b895dc9253b679be1105ca5cebea15743: Status 404 returned error can't find the container with id 38ef41beb82b0f00576ebe36ec4ae79b895dc9253b679be1105ca5cebea15743
Sep 18 22:55:49.411992 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:49.411970    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:49.607287 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:49.607236    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:55:50.031836 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:50.031798    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" event={"ID":"492e9ed3-fbb3-493e-a85e-030b09caf8e9","Type":"ContainerStarted","Data":"2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41"}
Sep 18 22:55:50.032252 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:50.031862    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" event={"ID":"492e9ed3-fbb3-493e-a85e-030b09caf8e9","Type":"ContainerStarted","Data":"38ef41beb82b0f00576ebe36ec4ae79b895dc9253b679be1105ca5cebea15743"}
Sep 18 22:55:52.043034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:52.043002    2564 generic.go:356] "Generic (PLEG): container finished" podID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerID="2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41" exitCode=0
Sep 18 22:55:52.043447 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:52.043040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" event={"ID":"492e9ed3-fbb3-493e-a85e-030b09caf8e9","Type":"ContainerDied","Data":"2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41"}
Sep 18 22:55:53.050952 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:53.050923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" event={"ID":"492e9ed3-fbb3-493e-a85e-030b09caf8e9","Type":"ContainerStarted","Data":"937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a"}
Sep 18 22:55:53.077106 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:53.077058    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podStartSLOduration=2.950846284 podStartE2EDuration="5.077043042s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.126196758s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:55:53.07575409 +0000 UTC m=+3931.932409741" watchObservedRunningTime="2026-09-18 22:55:53.077043042 +0000 UTC m=+3931.933698693"
Sep 18 22:55:55.613925 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:55.613894    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:55:55.614337 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:55:55.614143    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:55:59.106524 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:59.106489    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:59.106912 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:59.106553    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:55:59.107852 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:55:59.107822    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:56:01.617189 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:01.617163    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:56:01.617621 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:01.617362    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:56:07.061246 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:07.061195    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:07.161992 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:07.161957    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:08.613393 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:08.613363    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:56:08.613769 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:08.613552    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:56:09.107161 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:09.107115    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:56:13.147417 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:13.147377    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:13.614411 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:13.614379    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:56:13.614615 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:13.614572    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:56:14.304493 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:14.304442    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:14.749820 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:14.749713    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:19.107402 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:19.107358    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:56:19.833665 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:19.833628    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:19.932303 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:19.932251    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:23.613958 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:23.613929    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:56:23.614363 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:23.614101    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:56:25.613972 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:25.613938    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:56:25.614404 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:25.614209    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:56:29.107034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:29.106992    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:56:33.209581 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:33.209550    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:33.380958 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:33.380919    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:33.482461 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:33.482348    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:33.716825 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:33.716792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:33.836343 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:33.836308    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:33.930834 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:33.930800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:34.614126 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:34.614095    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:56:34.614733 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:34.614313    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:56:39.106619 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:39.106547    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:56:39.613939 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:39.613912    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:56:39.614135 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:39.614073    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:56:42.315000 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:42.314967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:56:45.613755 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:45.613722    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:56:45.614152 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:45.613935    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:56:49.106914 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:49.106868    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:56:50.614230 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:50.614202    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:56:50.614689 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:50.614455    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:56:58.613481 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:58.613451    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:56:58.613873 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:56:58.613649    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:56:59.106889 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:56:59.106849    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:57:01.615948 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:01.615910    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:57:01.616368 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:57:01.616132    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:57:09.106645 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:09.106596    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:57:13.613287 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:13.613245    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:57:13.613704 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:57:13.613463    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:57:16.613647 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:16.613615    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:57:16.614055 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:57:16.613843    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:57:19.106475 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:19.106429    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:57:26.614652 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:26.614610    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:57:26.615034 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:57:26.614771    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:57:27.614425 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:27.614395    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:57:27.614598 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:57:27.614581    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:57:29.107322 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:29.107275    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:57:36.336771 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:36.336713    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:57:37.444487 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:37.444455    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:57:39.107172 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:39.107124    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 18 22:57:39.619660 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:39.619631    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:57:39.619878 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:57:39.619856    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:57:41.620344 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:41.620309    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:57:41.620900 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:57:41.620633    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:57:44.082465 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:44.082390    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:57:49.116800 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:49.116768    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:57:49.124442 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:49.124417    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:57:53.613860 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:53.613825    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:57:53.614259 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:53.614009    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:57:53.614259 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:57:53.614044    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:57:53.614259 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:57:53.614183    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:57:57.855588 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:57.855551    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"]
Sep 18 22:57:57.860828 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:57.860800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:57:58.032169 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:57:58.032133    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:08.613581 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:08.613551    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:58:08.616008 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:08.613684    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:58:08.616008 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:08.613728    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:58:08.616008 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:08.613849    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:58:12.878556 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:12.878491    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main" containerID="cri-o://937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a" gracePeriod=45
Sep 18 22:58:13.561004 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.560978    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:58:13.714160 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714067    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-model-cache\") pod \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") "
Sep 18 22:58:13.714160 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714108    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-dshm\") pod \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") "
Sep 18 22:58:13.714160 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714143    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzg5p\" (UniqueName: \"kubernetes.io/projected/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kube-api-access-mzg5p\") pod \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") "
Sep 18 22:58:13.714488 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714187    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kserve-provision-location\") pod \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") "
Sep 18 22:58:13.714488 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714213    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/492e9ed3-fbb3-493e-a85e-030b09caf8e9-cabundle-cert\") pod \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") "
Sep 18 22:58:13.714488 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714376    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "492e9ed3-fbb3-493e-a85e-030b09caf8e9" (UID: "492e9ed3-fbb3-493e-a85e-030b09caf8e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.714488 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714390    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-home\") pod \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") "
Sep 18 22:58:13.714488 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714421    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tmp-dir\") pod \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") "
Sep 18 22:58:13.714756 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714553    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tls-certs\") pod \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\" (UID: \"492e9ed3-fbb3-493e-a85e-030b09caf8e9\") "
Sep 18 22:58:13.714756 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714712    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492e9ed3-fbb3-493e-a85e-030b09caf8e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "492e9ed3-fbb3-493e-a85e-030b09caf8e9" (UID: "492e9ed3-fbb3-493e-a85e-030b09caf8e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:58:13.714895 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714875    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/492e9ed3-fbb3-493e-a85e-030b09caf8e9-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.714955 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.714901    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.715021 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.715003    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-home" (OuterVolumeSpecName: "home") pod "492e9ed3-fbb3-493e-a85e-030b09caf8e9" (UID: "492e9ed3-fbb3-493e-a85e-030b09caf8e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.716661 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.716619    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kube-api-access-mzg5p" (OuterVolumeSpecName: "kube-api-access-mzg5p") pod "492e9ed3-fbb3-493e-a85e-030b09caf8e9" (UID: "492e9ed3-fbb3-493e-a85e-030b09caf8e9"). InnerVolumeSpecName "kube-api-access-mzg5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:58:13.716797 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.716734    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "492e9ed3-fbb3-493e-a85e-030b09caf8e9" (UID: "492e9ed3-fbb3-493e-a85e-030b09caf8e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:58:13.716844 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.716805    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-dshm" (OuterVolumeSpecName: "dshm") pod "492e9ed3-fbb3-493e-a85e-030b09caf8e9" (UID: "492e9ed3-fbb3-493e-a85e-030b09caf8e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.733075 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.733047    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "492e9ed3-fbb3-493e-a85e-030b09caf8e9" (UID: "492e9ed3-fbb3-493e-a85e-030b09caf8e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.739307 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.739287    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "492e9ed3-fbb3-493e-a85e-030b09caf8e9" (UID: "492e9ed3-fbb3-493e-a85e-030b09caf8e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.757869 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.757839    2564 generic.go:356] "Generic (PLEG): container finished" podID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerID="937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a" exitCode=0
Sep 18 22:58:13.757954 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.757897    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"
Sep 18 22:58:13.757954 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.757924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" event={"ID":"492e9ed3-fbb3-493e-a85e-030b09caf8e9","Type":"ContainerDied","Data":"937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a"}
Sep 18 22:58:13.758034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.757974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99" event={"ID":"492e9ed3-fbb3-493e-a85e-030b09caf8e9","Type":"ContainerDied","Data":"38ef41beb82b0f00576ebe36ec4ae79b895dc9253b679be1105ca5cebea15743"}
Sep 18 22:58:13.758034 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.757995    2564 scope.go:117] "RemoveContainer" containerID="937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a"
Sep 18 22:58:13.766786 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.766770    2564 scope.go:117] "RemoveContainer" containerID="2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41"
Sep 18 22:58:13.774653 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.774637    2564 scope.go:117] "RemoveContainer" containerID="937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a"
Sep 18 22:58:13.774887 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:13.774871    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a\": container with ID starting with 937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a not found: ID does not exist" containerID="937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a"
Sep 18 22:58:13.774931 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.774895    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a"} err="failed to get container status \"937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a\": rpc error: code = NotFound desc = could not find container \"937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a\": container with ID starting with 937b77f0e9c4b1d84bdbfd14a5cc023dd8e3b097219b4a2b04136d6b2cf3025a not found: ID does not exist"
Sep 18 22:58:13.774931 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.774910    2564 scope.go:117] "RemoveContainer" containerID="2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41"
Sep 18 22:58:13.775160 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:13.775143    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41\": container with ID starting with 2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41 not found: ID does not exist" containerID="2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41"
Sep 18 22:58:13.775212 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.775163    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41"} err="failed to get container status \"2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41\": rpc error: code = NotFound desc = could not find container \"2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41\": container with ID starting with 2792593604d2932a8947e8182854874e706dd08c749d2ca0c382a6fa18751d41 not found: ID does not exist"
Sep 18 22:58:13.779943 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.779917    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"]
Sep 18 22:58:13.786073 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.786046    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6467687bcb-fjd99"]
Sep 18 22:58:13.815667 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.815650    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.815731 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.815671    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.815731 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.815679    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492e9ed3-fbb3-493e-a85e-030b09caf8e9-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.815731 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.815687    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.815731 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.815694    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzg5p\" (UniqueName: \"kubernetes.io/projected/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kube-api-access-mzg5p\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.815731 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:13.815705    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/492e9ed3-fbb3-493e-a85e-030b09caf8e9-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 22:58:15.619335 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:15.619303    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" path="/var/lib/kubelet/pods/492e9ed3-fbb3-493e-a85e-030b09caf8e9/volumes"
Sep 18 22:58:19.613888 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:19.613857    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:58:19.614285 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:19.614088    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:58:22.613365 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:22.613332    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:58:22.613757 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:22.613547    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:58:26.384349 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:26.384308    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:26.480246 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:26.480209    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:32.706695 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:32.706664    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:33.613513 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:33.613483    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:58:33.613693 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:33.613662    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:58:33.958546 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:33.958459    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:34.164477 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:34.164439    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:36.613612 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:36.613583    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:58:36.613968 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:36.613784    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:58:38.577632 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:38.577581    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:38.653542 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:38.653510    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:45.727572 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:45.727536    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:47.614090 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:47.614056    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:58:47.614521 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:47.614243    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:58:47.976359 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:47.976278    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:48.073691 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:48.073653    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:48.613544 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:48.613514    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:58:48.613773 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:48.613750    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:58:53.024393 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:53.024357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:58:59.613502 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.613471    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:58:59.613870 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:58:59.613646    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:58:59.913905 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.913834    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"]
Sep 18 22:58:59.914379 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.914351    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="storage-initializer"
Sep 18 22:58:59.914379 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.914377    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="storage-initializer"
Sep 18 22:58:59.914582 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.914432    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main"
Sep 18 22:58:59.914582 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.914443    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main"
Sep 18 22:58:59.914582 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.914526    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="492e9ed3-fbb3-493e-a85e-030b09caf8e9" containerName="main"
Sep 18 22:58:59.917453 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.917433    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:58:59.920362 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.920340    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:58:59.920477 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.920344    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 18 22:58:59.920477 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.920403    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 18 22:58:59.920477 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.920421    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 18 22:58:59.920659 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.920588    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-ppbrm\""
Sep 18 22:58:59.920659 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.920602    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 18 22:58:59.931883 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.931858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"]
Sep 18 22:58:59.942107 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:58:59.942083    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:59:00.033979 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.033941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-524hg\" (UniqueName: \"kubernetes.io/projected/7f572109-6269-47c3-88cf-b99a95c7d0cf-kube-api-access-524hg\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.034161 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.034044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.034161 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.034104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f572109-6269-47c3-88cf-b99a95c7d0cf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.034287 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.034176    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.034287 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.034199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.034287 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.034220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.034287 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.034250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f572109-6269-47c3-88cf-b99a95c7d0cf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.034486 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.034348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135225 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135185    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135410 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f572109-6269-47c3-88cf-b99a95c7d0cf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135410 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135410 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-524hg\" (UniqueName: \"kubernetes.io/projected/7f572109-6269-47c3-88cf-b99a95c7d0cf-kube-api-access-524hg\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135576 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135576 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f572109-6269-47c3-88cf-b99a95c7d0cf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135576 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135576 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135516    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135771 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135771 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135707    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135872 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.135982 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.135961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.136170 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.136151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f572109-6269-47c3-88cf-b99a95c7d0cf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.137558 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.137540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.137966 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.137945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f572109-6269-47c3-88cf-b99a95c7d0cf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.144877 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.144853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-524hg\" (UniqueName: \"kubernetes.io/projected/7f572109-6269-47c3-88cf-b99a95c7d0cf-kube-api-access-524hg\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.236060 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.235963    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:00.517180 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.517154    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"]
Sep 18 22:59:00.518830 ip-10-0-130-79 kubenswrapper[2564]: W0918 22:59:00.518807    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f572109_6269_47c3_88cf_b99a95c7d0cf.slice/crio-9431cd65a5c44aede130b9a5ef44db844a832c869926c85e06c699bb435778bd WatchSource:0}: Error finding container 9431cd65a5c44aede130b9a5ef44db844a832c869926c85e06c699bb435778bd: Status 404 returned error can't find the container with id 9431cd65a5c44aede130b9a5ef44db844a832c869926c85e06c699bb435778bd
Sep 18 22:59:00.982222 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.982176    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" event={"ID":"7f572109-6269-47c3-88cf-b99a95c7d0cf","Type":"ContainerStarted","Data":"20ae50211f977b27ec3ae646d3206bec06e6f4848c60dc3d7ffa72f9e59ed07c"}
Sep 18 22:59:00.982710 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:00.982253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" event={"ID":"7f572109-6269-47c3-88cf-b99a95c7d0cf","Type":"ContainerStarted","Data":"9431cd65a5c44aede130b9a5ef44db844a832c869926c85e06c699bb435778bd"}
Sep 18 22:59:02.613632 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:02.613605    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:59:02.614012 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:02.613788    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:59:02.993047 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:02.992974    2564 generic.go:356] "Generic (PLEG): container finished" podID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerID="20ae50211f977b27ec3ae646d3206bec06e6f4848c60dc3d7ffa72f9e59ed07c" exitCode=0
Sep 18 22:59:02.993201 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:02.993053    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" event={"ID":"7f572109-6269-47c3-88cf-b99a95c7d0cf","Type":"ContainerDied","Data":"20ae50211f977b27ec3ae646d3206bec06e6f4848c60dc3d7ffa72f9e59ed07c"}
Sep 18 22:59:04.000192 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:04.000155    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" event={"ID":"7f572109-6269-47c3-88cf-b99a95c7d0cf","Type":"ContainerStarted","Data":"7f6f3ae76fc6e2fafa1b729150e398eaa629a35aa0c697f7e97a6f70b0f7797e"}
Sep 18 22:59:04.025971 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:04.025910    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podStartSLOduration=2.789986427 podStartE2EDuration="5.025889254s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.235902827s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:59:04.024112952 +0000 UTC m=+4122.880768605" watchObservedRunningTime="2026-09-18 22:59:04.025889254 +0000 UTC m=+4122.882544907"
Sep 18 22:59:04.236535 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:04.236495    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:10.236804 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:10.236754    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 22:59:10.238447 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:10.238415    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 22:59:10.613478 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:10.613448    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:59:10.613663 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:10.613643    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:59:13.613657 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:13.613625    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:59:13.614161 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:13.613811    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:59:14.805304 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:14.805247    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:59:16.015640 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:16.015600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:59:20.237051 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:20.237000    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 22:59:21.617699 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:21.617668    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:59:21.618079 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:21.617903    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:59:21.772455 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:21.772419    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:59:25.613617 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:25.613585    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:59:25.614051 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:25.613879    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:59:30.236869 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:30.236821    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 22:59:34.613993 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:34.613957    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:59:34.614382 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:34.614133    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:59:40.236682 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:40.236634    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 22:59:40.614712 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:40.614685    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:59:40.993490 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:40.993456    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:59:41.194406 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:41.194375    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/15.log"
Sep 18 22:59:41.194871 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:41.194855    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/14.log"
Sep 18 22:59:41.194951 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:41.194896    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970" exitCode=1
Sep 18 22:59:41.194998 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:41.194975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"}
Sep 18 22:59:41.195052 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:41.195042    2564 scope.go:117] "RemoveContainer" containerID="37dda8329c582873494fe3b65cb940a8af21a7669ad64a2bcaaa69e9fc600e50"
Sep 18 22:59:41.195390 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:41.195370    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 22:59:41.195617 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:41.195594    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:59:42.201457 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:42.201420    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/15.log"
Sep 18 22:59:44.179113 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:44.179080    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:59:44.179525 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:44.179220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 22:59:44.179568 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:44.179548    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 22:59:44.179764 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:44.179744    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:59:44.212506 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:44.212477    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 22:59:44.212723 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:44.212700    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 22:59:45.310568 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:45.310538    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:59:45.379875 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:45.379845    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 22:59:46.614307 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:46.614277    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 22:59:46.614682 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:46.614504    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 22:59:50.236860 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:50.236819    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 22:59:55.614073 ip-10-0-130-79 kubenswrapper[2564]: I0918 22:59:55.614038    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 22:59:55.614602 ip-10-0-130-79 kubenswrapper[2564]: E0918 22:59:55.614221    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:00:00.236383 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:00.236338    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 23:00:00.613283 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:00.613242    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 23:00:00.613497 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:00.613477    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:00:10.237078 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:10.237033    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 23:00:10.613631 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:10.613599    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:00:10.613821 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:10.613799    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:00:11.308639 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:11.308597    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:00:11.618015 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:11.617935    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 23:00:11.618180 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:11.618130    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:00:20.237432 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:20.237386    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 23:00:21.960801 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:21.960771    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/11.log"
Sep 18 23:00:21.961542 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:21.961518    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/15.log"
Sep 18 23:00:21.966763 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:21.966746    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:00:21.977406 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:21.977385    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/11.log"
Sep 18 23:00:21.978127 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:21.978109    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/15.log"
Sep 18 23:00:21.983301 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:21.983279    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:00:23.614097 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:23.614069    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:00:23.614592 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:23.614305    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:00:26.614446 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:26.614414    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 23:00:26.615054 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:26.614618    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:00:30.237172 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:30.237120    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 23:00:35.614533 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:35.614500    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:00:35.615041 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:35.614703    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:00:35.959801 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:35.959700    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:00:36.019642 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:36.019604    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:00:36.158282 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:36.158227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:00:38.614234 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:38.614204    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 23:00:39.500040 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:39.500011    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/12.log"
Sep 18 23:00:39.500521 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:39.500502    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/11.log"
Sep 18 23:00:39.500640 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:39.500543    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734" exitCode=1
Sep 18 23:00:39.500640 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:39.500610    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"}
Sep 18 23:00:39.500757 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:39.500672    2564 scope.go:117] "RemoveContainer" containerID="f5e95de95157c7317fb808c35cfacca7c5c0c5d331de02b480b61acd588db567"
Sep 18 23:00:39.500964 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:39.500944    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:00:39.501197 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:39.501176    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:00:40.236916 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:40.236865    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 23:00:40.507759 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:40.507688    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/12.log"
Sep 18 23:00:41.727321 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:41.727261    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:00:41.727843 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:41.727345    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:00:41.727843 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:41.727661    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:00:41.728025 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:41.727847    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:00:49.614239 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:49.614153    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:00:49.614625 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:49.614447    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:00:50.236479 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:50.236432    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 23:00:54.614008 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:54.613978    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:00:54.614381 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:00:54.614159    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:00:56.120890 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:56.120854    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:00:56.292523 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:00:56.292489    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:00.246558 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:00.246517    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 23:01:00.254179 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:00.254149    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 23:01:00.613564 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:00.613537    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:01:00.613737 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:00.613693    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:01:05.259200 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:05.259138    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:09.613891 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:09.613858    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:01:09.614390 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:09.614189    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:01:11.615649 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:11.615619    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:01:11.616111 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:11.615923    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:01:13.027275 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:13.027233    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"]
Sep 18 23:01:13.044609 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:13.044578    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:20.613810 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:20.613783    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:01:20.614260 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:20.614017    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:01:22.613774 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:22.613744    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:01:22.614153 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:22.613935    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:01:28.050871 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.050802    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" containerID="cri-o://7f6f3ae76fc6e2fafa1b729150e398eaa629a35aa0c697f7e97a6f70b0f7797e" gracePeriod=45
Sep 18 23:01:28.247819 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.247777    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 23:01:28.753443 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.753412    2564 generic.go:356] "Generic (PLEG): container finished" podID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerID="7f6f3ae76fc6e2fafa1b729150e398eaa629a35aa0c697f7e97a6f70b0f7797e" exitCode=0
Sep 18 23:01:28.753590 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.753485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" event={"ID":"7f572109-6269-47c3-88cf-b99a95c7d0cf","Type":"ContainerDied","Data":"7f6f3ae76fc6e2fafa1b729150e398eaa629a35aa0c697f7e97a6f70b0f7797e"}
Sep 18 23:01:28.801715 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.801691    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 23:01:28.819475 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.819446    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-tmp-dir\") pod \"7f572109-6269-47c3-88cf-b99a95c7d0cf\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") "
Sep 18 23:01:28.819598 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.819500    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-home\") pod \"7f572109-6269-47c3-88cf-b99a95c7d0cf\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") "
Sep 18 23:01:28.819598 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.819528    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f572109-6269-47c3-88cf-b99a95c7d0cf-cabundle-cert\") pod \"7f572109-6269-47c3-88cf-b99a95c7d0cf\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") "
Sep 18 23:01:28.819598 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.819549    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-dshm\") pod \"7f572109-6269-47c3-88cf-b99a95c7d0cf\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") "
Sep 18 23:01:28.819598 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.819597    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f572109-6269-47c3-88cf-b99a95c7d0cf-tls-certs\") pod \"7f572109-6269-47c3-88cf-b99a95c7d0cf\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") "
Sep 18 23:01:28.819824 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.819622    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-524hg\" (UniqueName: \"kubernetes.io/projected/7f572109-6269-47c3-88cf-b99a95c7d0cf-kube-api-access-524hg\") pod \"7f572109-6269-47c3-88cf-b99a95c7d0cf\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") "
Sep 18 23:01:28.819824 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.819658    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-kserve-provision-location\") pod \"7f572109-6269-47c3-88cf-b99a95c7d0cf\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") "
Sep 18 23:01:28.819824 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.819684    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-model-cache\") pod \"7f572109-6269-47c3-88cf-b99a95c7d0cf\" (UID: \"7f572109-6269-47c3-88cf-b99a95c7d0cf\") "
Sep 18 23:01:28.820031 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.819924    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f572109-6269-47c3-88cf-b99a95c7d0cf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f572109-6269-47c3-88cf-b99a95c7d0cf" (UID: "7f572109-6269-47c3-88cf-b99a95c7d0cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:01:28.820185 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.820157    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f572109-6269-47c3-88cf-b99a95c7d0cf" (UID: "7f572109-6269-47c3-88cf-b99a95c7d0cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:01:28.820305 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.820251    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-home" (OuterVolumeSpecName: "home") pod "7f572109-6269-47c3-88cf-b99a95c7d0cf" (UID: "7f572109-6269-47c3-88cf-b99a95c7d0cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:01:28.822005 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.821977    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f572109-6269-47c3-88cf-b99a95c7d0cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f572109-6269-47c3-88cf-b99a95c7d0cf" (UID: "7f572109-6269-47c3-88cf-b99a95c7d0cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:01:28.822245 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.822224    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f572109-6269-47c3-88cf-b99a95c7d0cf-kube-api-access-524hg" (OuterVolumeSpecName: "kube-api-access-524hg") pod "7f572109-6269-47c3-88cf-b99a95c7d0cf" (UID: "7f572109-6269-47c3-88cf-b99a95c7d0cf"). InnerVolumeSpecName "kube-api-access-524hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:01:28.822245 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.822236    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-dshm" (OuterVolumeSpecName: "dshm") pod "7f572109-6269-47c3-88cf-b99a95c7d0cf" (UID: "7f572109-6269-47c3-88cf-b99a95c7d0cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:01:28.838295 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.838226    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f572109-6269-47c3-88cf-b99a95c7d0cf" (UID: "7f572109-6269-47c3-88cf-b99a95c7d0cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:01:28.850828 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.850786    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f572109-6269-47c3-88cf-b99a95c7d0cf" (UID: "7f572109-6269-47c3-88cf-b99a95c7d0cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:01:28.921224 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.921188    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:01:28.921224 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.921217    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:01:28.921224 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.921226    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f572109-6269-47c3-88cf-b99a95c7d0cf-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:01:28.921224 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.921234    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:01:28.921500 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.921241    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f572109-6269-47c3-88cf-b99a95c7d0cf-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:01:28.921500 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.921249    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-524hg\" (UniqueName: \"kubernetes.io/projected/7f572109-6269-47c3-88cf-b99a95c7d0cf-kube-api-access-524hg\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:01:28.921500 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.921257    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:01:28.921500 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:28.921282    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f572109-6269-47c3-88cf-b99a95c7d0cf-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:01:29.760739 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:29.760718    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"
Sep 18 23:01:29.761119 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:29.760711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2" event={"ID":"7f572109-6269-47c3-88cf-b99a95c7d0cf","Type":"ContainerDied","Data":"9431cd65a5c44aede130b9a5ef44db844a832c869926c85e06c699bb435778bd"}
Sep 18 23:01:29.761119 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:29.760856    2564 scope.go:117] "RemoveContainer" containerID="7f6f3ae76fc6e2fafa1b729150e398eaa629a35aa0c697f7e97a6f70b0f7797e"
Sep 18 23:01:29.769569 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:29.769552    2564 scope.go:117] "RemoveContainer" containerID="20ae50211f977b27ec3ae646d3206bec06e6f4848c60dc3d7ffa72f9e59ed07c"
Sep 18 23:01:29.783130 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:29.783107    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"]
Sep 18 23:01:29.788896 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:29.788876    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b749f865f7t2k2"]
Sep 18 23:01:31.615532 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:31.615501    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:01:31.616063 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:31.615678    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:01:31.618912 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:31.618886    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" path="/var/lib/kubelet/pods/7f572109-6269-47c3-88cf-b99a95c7d0cf/volumes"
Sep 18 23:01:32.646089 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:32.646058    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:32.746507 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:32.746475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:32.846958 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:32.846927    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:34.613205 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:34.613179    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:01:34.613591 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:34.613371    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:01:39.097885 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:39.097846    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:43.613578 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:43.613546    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:01:43.613964 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:43.613747    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:01:44.752857 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:44.752825    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:44.905592 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:44.905563    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:45.051924 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.051891    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:45.085221 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.085189    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 23:01:45.085686 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.085669    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main"
Sep 18 23:01:45.085745 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.085688    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main"
Sep 18 23:01:45.085745 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.085703    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="storage-initializer"
Sep 18 23:01:45.085745 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.085709    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="storage-initializer"
Sep 18 23:01:45.085854 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.085763    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f572109-6269-47c3-88cf-b99a95c7d0cf" containerName="main"
Sep 18 23:01:45.088867 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.088849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.091455 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.091436    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 23:01:45.091558 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.091439    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:01:45.091558 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.091473    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 23:01:45.092109 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.092085    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 23:01:45.092236 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.092130    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-bjm8j\""
Sep 18 23:01:45.092236 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.092180    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-82vx4\""
Sep 18 23:01:45.092363 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.092282    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 23:01:45.108744 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.108724    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 23:01:45.137369 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.137341    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:45.166596 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.166570    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.166719 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.166632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.166719 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.166674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.166719 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.166695    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kkmx\" (UniqueName: \"kubernetes.io/projected/02640829-c0df-4e04-a087-c43cab1b1d22-kube-api-access-2kkmx\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.166830 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.166742    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.166830 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.166777    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.166830 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.166808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02640829-c0df-4e04-a087-c43cab1b1d22-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.166931 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.166829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02640829-c0df-4e04-a087-c43cab1b1d22-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.267498 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267466    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.267669 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267504    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.267669 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kkmx\" (UniqueName: \"kubernetes.io/projected/02640829-c0df-4e04-a087-c43cab1b1d22-kube-api-access-2kkmx\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.267803 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.267803 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.267803 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02640829-c0df-4e04-a087-c43cab1b1d22-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.267803 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02640829-c0df-4e04-a087-c43cab1b1d22-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.268002 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.268002 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267795    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.268002 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.268002 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.267987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.268149 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.268116    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.268343 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.268322    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02640829-c0df-4e04-a087-c43cab1b1d22-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.269840 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.269820    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.270124 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.270103    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02640829-c0df-4e04-a087-c43cab1b1d22-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.276082 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.276062    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kkmx\" (UniqueName: \"kubernetes.io/projected/02640829-c0df-4e04-a087-c43cab1b1d22-kube-api-access-2kkmx\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.400012 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.399948    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:45.587261 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.587208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:45.613945 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.613922    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:01:45.614111 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:45.614092    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:01:45.663119 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.663015    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:45.681411 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.681392    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 23:01:45.683126 ip-10-0-130-79 kubenswrapper[2564]: W0918 23:01:45.683104    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02640829_c0df_4e04_a087_c43cab1b1d22.slice/crio-ca817e06478f6fefb28a44ad0390d22d4ae4b4efc4180084ae1785c7df5a8b7d WatchSource:0}: Error finding container ca817e06478f6fefb28a44ad0390d22d4ae4b4efc4180084ae1785c7df5a8b7d: Status 404 returned error can't find the container with id ca817e06478f6fefb28a44ad0390d22d4ae4b4efc4180084ae1785c7df5a8b7d
Sep 18 23:01:45.839503 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.839463    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"02640829-c0df-4e04-a087-c43cab1b1d22","Type":"ContainerStarted","Data":"3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe"}
Sep 18 23:01:45.839846 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:45.839521    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"02640829-c0df-4e04-a087-c43cab1b1d22","Type":"ContainerStarted","Data":"ca817e06478f6fefb28a44ad0390d22d4ae4b4efc4180084ae1785c7df5a8b7d"}
Sep 18 23:01:47.593333 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:47.593291    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:48.857728 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:48.857694    2564 generic.go:356] "Generic (PLEG): container finished" podID="02640829-c0df-4e04-a087-c43cab1b1d22" containerID="3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe" exitCode=0
Sep 18 23:01:48.858225 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:48.857771    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"02640829-c0df-4e04-a087-c43cab1b1d22","Type":"ContainerDied","Data":"3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe"}
Sep 18 23:01:49.865553 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:49.865510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"02640829-c0df-4e04-a087-c43cab1b1d22","Type":"ContainerStarted","Data":"5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6"}
Sep 18 23:01:49.888696 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:49.888641    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.605898887 podStartE2EDuration="4.888624783s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.282725896s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:01:49.888255509 +0000 UTC m=+4288.744911174" watchObservedRunningTime="2026-09-18 23:01:49.888624783 +0000 UTC m=+4288.745280435"
Sep 18 23:01:50.400472 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:50.400439    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:54.614396 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:54.614363    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:01:54.614766 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:54.614631    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:01:55.400201 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:55.400165    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:01:55.401805 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:55.401776    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 23:01:58.628841 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:58.628809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:01:59.613467 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:01:59.613440    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:01:59.613666 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:01:59.613623    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:02:02.244408 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:02.244363    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:02:05.401404 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:05.401353    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 23:02:09.613821 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:09.613791    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:02:09.614207 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:02:09.613952    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:02:13.614258 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:13.614216    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:02:13.614748 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:02:13.614498    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:02:15.401022 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:15.400963    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 23:02:20.614168 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:20.614139    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:02:20.614571 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:02:20.614378    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:02:25.401220 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:25.401176    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 23:02:26.613467 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:26.613428    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:02:26.613946 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:02:26.613703    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:02:35.400655 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:35.400613    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 23:02:35.621668 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:35.621633    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:02:35.621991 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:02:35.621955    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:02:37.619494 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:37.619465    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:02:37.619881 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:02:37.619658    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:02:45.401070 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:45.400999    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 23:02:46.613726 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:46.613694    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:02:46.614123 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:02:46.613893    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:02:52.613555 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:52.613526    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:02:52.613923 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:02:52.613722    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:02:55.401125 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:55.401076    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 23:02:58.614366 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:02:58.614331    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:02:58.614727 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:02:58.614619    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:03:05.401203 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:05.401151    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 23:03:05.613611 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:05.613581    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:03:05.613864 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:05.613842    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:03:07.932829 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:07.932790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:03:09.294639 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:09.294602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:03:10.613783 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:10.613753    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:03:10.614176 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:10.613912    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:03:15.401591 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:15.401530    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 23:03:17.614438 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:17.614407    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:03:17.614825 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:17.614645    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:03:23.613288 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:23.613238    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:03:23.613753 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:23.613457    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:03:25.410240 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:25.410205    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:03:25.417787 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:25.417763    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:03:30.585771 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:30.585737    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 23:03:30.635934 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:30.635897    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:03:30.735508 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:30.735477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:03:31.620880 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:31.620848    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:03:31.621346 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:31.621022    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:03:35.613884 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:35.613850    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:03:35.614368 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:35.614050    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:03:40.276786 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:40.276754    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:03:42.613456 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:42.613427    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:03:42.613921 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:42.613688    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:03:45.608521 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:45.608413    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main" containerID="cri-o://5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6" gracePeriod=45
Sep 18 23:03:46.348428 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.348405    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:03:46.426198 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.426116    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02640829-c0df-4e04-a087-c43cab1b1d22-tls-certs\") pod \"02640829-c0df-4e04-a087-c43cab1b1d22\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") "
Sep 18 23:03:46.426198 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.426151    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-home\") pod \"02640829-c0df-4e04-a087-c43cab1b1d22\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") "
Sep 18 23:03:46.426443 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.426220    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-dshm\") pod \"02640829-c0df-4e04-a087-c43cab1b1d22\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") "
Sep 18 23:03:46.426443 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.426243    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-kserve-provision-location\") pod \"02640829-c0df-4e04-a087-c43cab1b1d22\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") "
Sep 18 23:03:46.426443 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.426332    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02640829-c0df-4e04-a087-c43cab1b1d22-cabundle-cert\") pod \"02640829-c0df-4e04-a087-c43cab1b1d22\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") "
Sep 18 23:03:46.426443 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.426367    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-tmp-dir\") pod \"02640829-c0df-4e04-a087-c43cab1b1d22\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") "
Sep 18 23:03:46.426443 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.426414    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-model-cache\") pod \"02640829-c0df-4e04-a087-c43cab1b1d22\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") "
Sep 18 23:03:46.426443 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.426441    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kkmx\" (UniqueName: \"kubernetes.io/projected/02640829-c0df-4e04-a087-c43cab1b1d22-kube-api-access-2kkmx\") pod \"02640829-c0df-4e04-a087-c43cab1b1d22\" (UID: \"02640829-c0df-4e04-a087-c43cab1b1d22\") "
Sep 18 23:03:46.426754 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.426711    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02640829-c0df-4e04-a087-c43cab1b1d22-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02640829-c0df-4e04-a087-c43cab1b1d22" (UID: "02640829-c0df-4e04-a087-c43cab1b1d22"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:03:46.427053 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.427014    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-model-cache" (OuterVolumeSpecName: "model-cache") pod "02640829-c0df-4e04-a087-c43cab1b1d22" (UID: "02640829-c0df-4e04-a087-c43cab1b1d22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:46.427053 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.427031    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-home" (OuterVolumeSpecName: "home") pod "02640829-c0df-4e04-a087-c43cab1b1d22" (UID: "02640829-c0df-4e04-a087-c43cab1b1d22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:46.428885 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.428858    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-dshm" (OuterVolumeSpecName: "dshm") pod "02640829-c0df-4e04-a087-c43cab1b1d22" (UID: "02640829-c0df-4e04-a087-c43cab1b1d22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:46.428885 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.428864    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02640829-c0df-4e04-a087-c43cab1b1d22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02640829-c0df-4e04-a087-c43cab1b1d22" (UID: "02640829-c0df-4e04-a087-c43cab1b1d22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:03:46.429022 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.428944    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02640829-c0df-4e04-a087-c43cab1b1d22-kube-api-access-2kkmx" (OuterVolumeSpecName: "kube-api-access-2kkmx") pod "02640829-c0df-4e04-a087-c43cab1b1d22" (UID: "02640829-c0df-4e04-a087-c43cab1b1d22"). InnerVolumeSpecName "kube-api-access-2kkmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:03:46.444475 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.444447    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02640829-c0df-4e04-a087-c43cab1b1d22" (UID: "02640829-c0df-4e04-a087-c43cab1b1d22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:46.457585 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.457554    2564 generic.go:356] "Generic (PLEG): container finished" podID="02640829-c0df-4e04-a087-c43cab1b1d22" containerID="5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6" exitCode=0
Sep 18 23:03:46.457688 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.457592    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"02640829-c0df-4e04-a087-c43cab1b1d22","Type":"ContainerDied","Data":"5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6"}
Sep 18 23:03:46.457688 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.457603    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 23:03:46.457688 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.457641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"02640829-c0df-4e04-a087-c43cab1b1d22","Type":"ContainerDied","Data":"ca817e06478f6fefb28a44ad0390d22d4ae4b4efc4180084ae1785c7df5a8b7d"}
Sep 18 23:03:46.457688 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.457668    2564 scope.go:117] "RemoveContainer" containerID="5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6"
Sep 18 23:03:46.458324 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.458304    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02640829-c0df-4e04-a087-c43cab1b1d22" (UID: "02640829-c0df-4e04-a087-c43cab1b1d22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:46.465974 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.465957    2564 scope.go:117] "RemoveContainer" containerID="3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe"
Sep 18 23:03:46.473675 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.473659    2564 scope.go:117] "RemoveContainer" containerID="5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6"
Sep 18 23:03:46.473929 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:46.473908    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6\": container with ID starting with 5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6 not found: ID does not exist" containerID="5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6"
Sep 18 23:03:46.474027 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.473935    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6"} err="failed to get container status \"5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6\": rpc error: code = NotFound desc = could not find container \"5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6\": container with ID starting with 5b776a24c728b20f4458262a5129aab9108d33d9322a380c3ecb9db81ed2c3b6 not found: ID does not exist"
Sep 18 23:03:46.474027 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.473954    2564 scope.go:117] "RemoveContainer" containerID="3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe"
Sep 18 23:03:46.474195 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:46.474178    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe\": container with ID starting with 3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe not found: ID does not exist" containerID="3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe"
Sep 18 23:03:46.474236 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.474204    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe"} err="failed to get container status \"3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe\": rpc error: code = NotFound desc = could not find container \"3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe\": container with ID starting with 3d65de8b68eeb55c676b295257ca3d006530774f57b5f2a85ed1c98dc7840dbe not found: ID does not exist"
Sep 18 23:03:46.527719 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.527693    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kkmx\" (UniqueName: \"kubernetes.io/projected/02640829-c0df-4e04-a087-c43cab1b1d22-kube-api-access-2kkmx\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:03:46.527719 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.527715    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02640829-c0df-4e04-a087-c43cab1b1d22-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:03:46.527883 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.527725    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:03:46.527883 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.527733    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:03:46.527883 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.527740    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:03:46.527883 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.527749    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02640829-c0df-4e04-a087-c43cab1b1d22-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:03:46.527883 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.527757    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:03:46.527883 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.527764    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02640829-c0df-4e04-a087-c43cab1b1d22-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:03:46.782972 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.782941    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 23:03:46.786340 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:46.786316    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 23:03:47.620082 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:47.620045    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" path="/var/lib/kubelet/pods/02640829-c0df-4e04-a087-c43cab1b1d22/volumes"
Sep 18 23:03:50.613930 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:50.613896    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:03:50.614336 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:50.614087    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:03:54.613714 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:54.613684    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:03:54.614082 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:03:54.613887    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:03:57.492588 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:57.492532    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:03:57.621743 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:57.621710    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:03:57.724804 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:03:57.724773    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:04.028221 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:04.028187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:04.613350 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:04.613323    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:04:04.613530 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:04.613510    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:04:08.613142 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:08.613116    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:04:08.613555 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:08.613332    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:04:09.901681 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:09.901647    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:09.999770 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:09.999737    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:17.619183 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:17.619157    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:04:17.619742 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:17.619409    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:04:17.912094 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:17.912015    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:20.449587 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:20.449537    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:20.543684 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:20.543655    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:21.617359 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:21.617330    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:04:21.617717 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:21.617568    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:04:27.057248 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:27.057215    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:28.613567 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:28.613530    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:04:28.613964 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:28.613747    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:04:33.362934 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:33.362878    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:33.430487 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:33.430441    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:33.613556 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:33.613491    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:04:33.613689 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:33.613670    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:04:36.392632 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:36.392585    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:36.493322 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:36.493283    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:42.613572 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:42.613542    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:04:42.613964 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:42.613727    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:04:45.613418 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:45.613388    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:04:45.733300 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:45.733261    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/16.log"
Sep 18 23:04:45.733715 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:45.733700    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/15.log"
Sep 18 23:04:45.733764 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:45.733735    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8" exitCode=1
Sep 18 23:04:45.733831 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:45.733809    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"}
Sep 18 23:04:45.733912 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:45.733874    2564 scope.go:117] "RemoveContainer" containerID="fa2944913ebef0aa5bf819c5824249e951c2bf2be6f89a7acd2cd7601143c970"
Sep 18 23:04:45.734167 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:45.734149    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:04:45.734426 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:45.734405    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:04:46.741277 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:46.741248    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/16.log"
Sep 18 23:04:47.942989 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:47.942958    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:48.246331 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:48.246236    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:04:54.179089 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:54.179048    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:04:54.179496 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:54.179110    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:04:54.179496 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:54.179426    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:04:54.179645 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:54.179623    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:04:56.613215 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:04:56.613179    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:04:56.613657 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:04:56.613456    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:05:09.613549 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:09.613518    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:05:09.614074 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:09.613572    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:05:09.614074 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:05:09.613730    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:05:09.614074 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:05:09.613770    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:05:12.121672 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:12.121639    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:05:21.914787 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:21.914706    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:05:21.992743 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:21.992715    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/12.log"
Sep 18 23:05:21.993535 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:21.993518    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/16.log"
Sep 18 23:05:21.998184 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:21.998164    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:05:22.011185 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:22.011153    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/12.log"
Sep 18 23:05:22.012032 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:22.012012    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/16.log"
Sep 18 23:05:22.012160 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:22.012130    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:05:22.017506 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:22.017489    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:05:24.614047 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:24.614013    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:05:24.614482 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:24.614184    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:05:24.614482 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:05:24.614290    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:05:24.614482 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:05:24.614362    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:05:38.613768 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:38.613743    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:05:38.614137 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:05:38.613900    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:05:39.614117 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:39.614088    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:05:39.614518 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:05:39.614303    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:05:49.619130 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:49.619101    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:05:50.050307 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:50.050259    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/13.log"
Sep 18 23:05:50.050768 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:50.050753    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/12.log"
Sep 18 23:05:50.050821 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:50.050789    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d" exitCode=1
Sep 18 23:05:50.050888 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:50.050867    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"}
Sep 18 23:05:50.050943 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:50.050926    2564 scope.go:117] "RemoveContainer" containerID="4836fe159a97c8de41fc9b4ccbe290892554c3848783867c780e3c151651b734"
Sep 18 23:05:50.051254 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:50.051237    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:05:50.051518 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:05:50.051492    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:05:51.057251 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:51.057219    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/13.log"
Sep 18 23:05:51.616230 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:51.616196    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:05:51.616519 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:05:51.616494    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:05:51.727925 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:51.727881    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:05:51.728092 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:51.727947    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:05:51.728232 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:05:51.728220    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:05:51.728452 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:05:51.728431    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:06:03.613373 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:03.613339    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:06:03.613790 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:03.613501    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:06:03.613790 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:03.613506    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:06:03.613790 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:03.613667    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:06:04.737548 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:04.737517    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:11.640042 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:11.640008    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:11.744393 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:11.744365    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:14.613594 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:14.613563    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:06:14.613990 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:14.613779    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:06:14.747019 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:14.746988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:14.847675 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:14.847646    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:16.533048 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:16.533013    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:16.614224 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:16.614195    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:06:16.614444 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:16.614425    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:06:16.884126 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:16.884012    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:18.598932 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:18.598893    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:24.233563 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:24.233528    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:24.297236 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:24.297208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:25.350113 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:25.350073    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:25.451101 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:25.451065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:25.548372 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:25.548342    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:25.613947 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:25.613849    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:06:25.614117 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:25.614086    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:06:25.675855 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:25.675825    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:27.614014 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:27.613985    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:06:27.614441 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:27.614182    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:06:29.920442 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:29.920414    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:31.238585 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:31.238547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:31.338028 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:31.337989    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:36.634104 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:36.634065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:38.613513 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:38.613484    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:06:38.613898 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:38.613554    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:06:38.613898 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:38.613692    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:06:38.613898 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:38.613705    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:06:51.619662 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:51.619581    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:06:51.620111 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:51.619800    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:06:51.620111 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:51.619846    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:06:51.620111 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:06:51.620007    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:06:57.437980 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:57.437943    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:06:57.495835 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:06:57.495802    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:07:02.613279 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:02.613247    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:07:02.613669 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:07:02.613449    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:07:06.613715 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:06.613686    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:07:06.614086 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:07:06.613893    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:07:15.620202 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:15.620175    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:07:15.620690 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:07:15.620373    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:07:20.614286 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:20.614239    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:07:20.614668 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:07:20.614471    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:07:27.614130 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:27.614091    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:07:27.614603 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:07:27.614401    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:07:30.307788 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:30.307749    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:07:31.615931 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:31.615899    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:07:31.616334 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:07:31.616129    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:07:38.613797 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:38.613763    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:07:38.614192 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:07:38.613965    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:07:46.613507 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:46.613478    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:07:46.613915 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:07:46.613681    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:07:47.104324 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:47.104231    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:07:47.203575 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:47.203539    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:07:53.519676 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:53.519632    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:07:53.614241 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:53.614206    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:07:53.614524 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:07:53.614497    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:07:54.740316 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:54.740259    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:07:55.065479 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:55.065445    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:07:59.162561 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:59.162529    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:07:59.267686 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:07:59.267652    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:08:01.069826 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:01.069779    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:08:01.615880 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:01.615846    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:08:01.616094 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:01.616042    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:08:04.613220 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:04.613188    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:08:04.613766 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:04.613480    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:08:13.613655 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:13.613625    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:08:13.616015 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:13.613822    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:08:19.614106 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:19.614069    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:08:19.619278 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:19.618013    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:08:25.614031 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:25.613986    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:08:25.614504 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:25.614258    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:08:28.930084 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:28.930047    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:08:30.028513 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:30.028480    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:08:32.613625 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:32.613596    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:08:32.614010 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:32.613786    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:08:38.529026 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:38.528994    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:08:38.613868 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:38.613831    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:08:38.614043 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:38.614008    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:08:43.613289 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:43.613233    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:08:43.613755 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:43.613511    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:08:49.613260 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:49.613229    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:08:49.613719 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:49.613436    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:08:55.614231 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:55.614200    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:08:55.614730 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:08:55.614434    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:08:58.614257 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:58.614222    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:08:58.714694 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:58.714662    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:08:58.814557 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:58.814526    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:08:58.911025 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:58.910942    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:08:58.973133 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:08:58.973106    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:01.217780 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:01.217751    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:01.297440 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:01.297414    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:02.051328 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:02.051294    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:04.613566 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:04.613534    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:09:04.613991 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:04.613726    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:09:04.624399 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:04.624377    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:04.689501 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:04.689478    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:06.613954 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:06.613927    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:09:06.614341 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:06.614119    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:09:15.613651 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:15.613620    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:09:15.614028 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:15.613824    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:09:15.703147 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:15.703114    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:17.613929 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:17.613898    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:09:17.614419 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:17.614107    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:09:18.493373 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:18.493340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:18.593597 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:18.593564    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:25.125007 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:25.124969    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:29.613706 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:29.613676    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:09:29.614191 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:29.613861    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:09:30.724405 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:30.724375    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:30.825013 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:30.824982    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:32.613852 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:32.613819    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:09:32.614251 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:32.614019    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:09:44.613612 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:44.613537    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:09:44.614046 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:44.613707    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:09:45.769035 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.768997    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"]
Sep 18 23:09:45.769417 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.769403    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main"
Sep 18 23:09:45.769417 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.769417    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main"
Sep 18 23:09:45.769490 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.769439    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="storage-initializer"
Sep 18 23:09:45.769490 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.769445    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="storage-initializer"
Sep 18 23:09:45.769553 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.769500    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02640829-c0df-4e04-a087-c43cab1b1d22" containerName="main"
Sep 18 23:09:45.772649 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.772629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.775807 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.775787    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Sep 18 23:09:45.776385 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.776365    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Sep 18 23:09:45.776730 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.776712    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:09:45.776807 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.776712    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-pdcfl\""
Sep 18 23:09:45.776872 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.776802    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 23:09:45.776872 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.776825    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Sep 18 23:09:45.787426 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.787401    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"]
Sep 18 23:09:45.809608 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.809585    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-model-cache\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.809708 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.809619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e70533-b9ac-4b8d-a073-838c050337d9-tls-certs\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.809708 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.809645    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.809708 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.809698    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-tmp-dir\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.809818 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.809728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kq8\" (UniqueName: \"kubernetes.io/projected/01e70533-b9ac-4b8d-a073-838c050337d9-kube-api-access-z8kq8\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.809818 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.809754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-dshm\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.809818 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.809796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e70533-b9ac-4b8d-a073-838c050337d9-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.809915 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.809832    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-home\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.812835 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.812798    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:45.910889 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.910856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-model-cache\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911054 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.910893    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e70533-b9ac-4b8d-a073-838c050337d9-tls-certs\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911054 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.910917    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911054 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-tmp-dir\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911233 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kq8\" (UniqueName: \"kubernetes.io/projected/01e70533-b9ac-4b8d-a073-838c050337d9-kube-api-access-z8kq8\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911233 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-dshm\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911233 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911145    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e70533-b9ac-4b8d-a073-838c050337d9-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911233 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911197    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-home\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911540 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-model-cache\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911540 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911415    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911540 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911478    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-tmp-dir\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911656 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-home\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.911943 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.911926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e70533-b9ac-4b8d-a073-838c050337d9-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.913381 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.913360    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-dshm\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.913668 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.913651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e70533-b9ac-4b8d-a073-838c050337d9-tls-certs\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:45.920008 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:45.919981    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kq8\" (UniqueName: \"kubernetes.io/projected/01e70533-b9ac-4b8d-a073-838c050337d9-kube-api-access-z8kq8\") pod \"tls-verification-test-kserve-prefill-7f8db478bb-9l47n\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:46.022906 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:46.022818    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:46.083708 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:46.083678    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:46.364673 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:46.364648    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"]
Sep 18 23:09:46.366411 ip-10-0-130-79 kubenswrapper[2564]: W0918 23:09:46.366382    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e70533_b9ac_4b8d_a073_838c050337d9.slice/crio-f1b50aed842fd9c294b11b34e44298f74a74c1989ff5ed105fc90391c6cc05be WatchSource:0}: Error finding container f1b50aed842fd9c294b11b34e44298f74a74c1989ff5ed105fc90391c6cc05be: Status 404 returned error can't find the container with id f1b50aed842fd9c294b11b34e44298f74a74c1989ff5ed105fc90391c6cc05be
Sep 18 23:09:47.193373 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:47.193330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" event={"ID":"01e70533-b9ac-4b8d-a073-838c050337d9","Type":"ContainerStarted","Data":"9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be"}
Sep 18 23:09:47.193984 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:47.193411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" event={"ID":"01e70533-b9ac-4b8d-a073-838c050337d9","Type":"ContainerStarted","Data":"f1b50aed842fd9c294b11b34e44298f74a74c1989ff5ed105fc90391c6cc05be"}
Sep 18 23:09:47.614214 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:47.614180    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:09:48.203501 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:48.203475    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/17.log"
Sep 18 23:09:48.203984 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:48.203966    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/16.log"
Sep 18 23:09:48.204067 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:48.204010    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32" exitCode=1
Sep 18 23:09:48.204122 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:48.204090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"}
Sep 18 23:09:48.204176 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:48.204152    2564 scope.go:117] "RemoveContainer" containerID="706c4439ce89df3f7d21881112c7f2f89223096d65c31ac83157002733a5e6e8"
Sep 18 23:09:48.204587 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:48.204566    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:09:48.204798 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:48.204778    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:09:49.210449 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:49.210415    2564 generic.go:356] "Generic (PLEG): container finished" podID="01e70533-b9ac-4b8d-a073-838c050337d9" containerID="9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be" exitCode=0
Sep 18 23:09:49.210937 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:49.210494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" event={"ID":"01e70533-b9ac-4b8d-a073-838c050337d9","Type":"ContainerDied","Data":"9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be"}
Sep 18 23:09:49.212164 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:49.212146    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/17.log"
Sep 18 23:09:50.218887 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:50.218853    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" event={"ID":"01e70533-b9ac-4b8d-a073-838c050337d9","Type":"ContainerStarted","Data":"fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda"}
Sep 18 23:09:50.243969 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:50.243905    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podStartSLOduration=2.96989313 podStartE2EDuration="5.243885716s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.273992586s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:09:50.242957163 +0000 UTC m=+4769.099612826" watchObservedRunningTime="2026-09-18 23:09:50.243885716 +0000 UTC m=+4769.100541363"
Sep 18 23:09:54.179313 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:54.179280    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:09:54.179695 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:54.179340    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:09:54.179734 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:54.179705    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:09:54.179940 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:54.179919    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:09:56.084200 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:56.084164    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:56.084591 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:56.084239    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:09:56.085894 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:56.085864    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:09:57.235409 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:57.235375    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:09:58.613243 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:09:58.613216    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:09:58.613637 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:09:58.613454    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:10:06.084466 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:06.084424    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:10:07.613364 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:07.613329    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:10:07.613750 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:10:07.613565    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:10:08.746448 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:08.746417    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:10:09.614757 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:09.614728    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:10:09.614929 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:10:09.614911    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:10:16.084973 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:16.084928    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:10:20.613971 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:20.613937    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:10:20.614446 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:10:20.614197    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:10:22.025067 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:22.025043    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/13.log"
Sep 18 23:10:22.025914 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:22.025897    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/17.log"
Sep 18 23:10:22.034760 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:22.034738    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:10:22.045602 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:22.045565    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/13.log"
Sep 18 23:10:22.046396 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:22.046378    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/17.log"
Sep 18 23:10:22.051610 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:22.051594    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:10:22.613837 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:22.613804    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:10:22.614106 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:10:22.614077    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:10:26.084910 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:26.084870    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:10:34.613835 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:34.613804    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:10:34.614214 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:10:34.613997    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:10:36.084592 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:36.084547    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:10:36.376434 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:36.376305    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:10:36.613974 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:36.613945    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:10:36.614175 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:10:36.614155    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:10:46.085082 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:46.085039    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:10:49.613680 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:49.613645    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:10:49.614293 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:49.613770    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:10:49.614293 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:10:49.613869    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:10:49.614293 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:10:49.613927    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:10:56.084710 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:10:56.084666    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:11:03.614438 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:03.614402    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:11:04.613427 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:04.613395    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:11:04.613652 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:04.613626    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:11:04.614194 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:04.614178    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/14.log"
Sep 18 23:11:04.614785 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:04.614758    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/13.log"
Sep 18 23:11:04.615084 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:04.614811    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149" exitCode=1
Sep 18 23:11:04.615084 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:04.614848    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"}
Sep 18 23:11:04.615084 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:04.614904    2564 scope.go:117] "RemoveContainer" containerID="31d9507c1019470354db08f7594915ebe066eb50367ec2d0cb69df12bd5e5f6d"
Sep 18 23:11:04.615223 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:04.615175    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:11:04.615391 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:04.615374    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:11:05.621135 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:05.621107    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/14.log"
Sep 18 23:11:06.084866 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:06.084805    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:11:11.728057 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:11.728014    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:11:11.728508 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:11.728077    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:11:11.728508 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:11.728421    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:11:11.728612 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:11.728593    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:11:16.084648 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:16.084555    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:11:19.613652 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:19.613621    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:11:19.614094 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:19.613861    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:11:24.613748 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:24.613714    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:11:24.614213 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:24.614020    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:11:26.084630 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:26.084585    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:11:30.613504 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:30.613468    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:11:30.613894 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:30.613739    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:11:36.085100 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:36.085046    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 18 23:11:37.613712 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:37.613684    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:11:37.614186 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:37.613884    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:11:43.613680 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:43.613646    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:11:43.614053 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:43.613832    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:11:46.094447 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:46.094416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:11:46.101982 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:46.101953    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:11:49.619036 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:49.619008    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:11:49.619423 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:49.619189    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:11:54.613738 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:11:54.613702    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:11:54.614203 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:11:54.613984    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:12:02.613231 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:02.613202    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:12:02.613778 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:12:02.613416    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:12:08.613989 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:08.613955    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:12:08.614439 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:12:08.614129    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:12:10.956633 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:10.956596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:13.619206 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:13.619173    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:12:13.619686 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:12:13.619453    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:12:23.614536 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:23.614505    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:12:23.614911 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:12:23.614731    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:12:28.613826 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:28.613794    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:12:28.614233 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:12:28.614025    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:12:31.236443 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:31.236403    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:31.335323 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:31.335292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:31.436064 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:31.436033    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:31.612368 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:31.612331    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:31.678825 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:31.678797    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:33.804805 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:33.804770    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:33.887492 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:33.887462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:34.626285 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:34.626237    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:36.408782 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:36.408755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:36.510280 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:36.510242    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:37.426016 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:37.425982    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:37.526132 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:37.526103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:38.613745 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:38.613717    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:12:38.614117 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:12:38.613885    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:12:39.560533 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:39.560503    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:39.633351 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:39.633316    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:12:43.613618 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:43.613592    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:12:43.614081 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:12:43.613782    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:12:50.613634 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:50.613602    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:12:50.614024 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:12:50.613789    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:12:56.614282 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:12:56.614245    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:12:56.614654 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:12:56.614435    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:13:00.670476 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:00.670445    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:13:04.613846 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:04.613818    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:13:04.614216 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:13:04.614005    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:13:07.613595 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:07.613558    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:13:07.614137 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:13:07.613808    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:13:18.613782 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:18.613753    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:13:18.616211 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:13:18.613941    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:13:21.617232 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:21.617200    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:13:21.617640 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:13:21.617423    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:13:31.617700 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:31.617671    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:13:31.618179 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:13:31.617909    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:13:36.614010 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:36.613978    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:13:36.614582 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:13:36.614229    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:13:46.613478 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:46.613448    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:13:46.613885 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:13:46.613661    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:13:47.613408 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:47.613376    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:13:47.613617 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:13:47.613598    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:13:57.613571 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:13:57.613541    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:13:57.613974 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:13:57.613718    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:14:02.613871 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:02.613844    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:14:02.614285 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:14:02.614044    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:14:09.613943 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:09.613913    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:14:09.614497 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:14:09.614139    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:14:14.613369 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:14.613297    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:14:14.613735 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:14:14.613492    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:14:21.617280 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:21.617238    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:14:21.617758 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:14:21.617460    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:14:26.613722 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:26.613690    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:14:26.614107 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:14:26.613898    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:14:34.613811 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:34.613778    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:14:34.614211 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:14:34.613951    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:14:39.613249 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:39.613214    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:14:39.613724 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:14:39.613453    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:14:40.266683 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:40.266649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:14:45.613789 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:45.613754    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:14:45.614202 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:14:45.613926    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:14:50.613485 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:50.613459    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:14:50.613848 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:14:50.613642    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:14:58.256022 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:14:58.255986    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:00.613619 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:00.613591    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:15:00.768957 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:00.768932    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/18.log"
Sep 18 23:15:00.769412 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:00.769392    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/17.log"
Sep 18 23:15:00.769529 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:00.769437    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0" exitCode=1
Sep 18 23:15:00.769600 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:00.769523    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"}
Sep 18 23:15:00.769654 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:00.769601    2564 scope.go:117] "RemoveContainer" containerID="f3e3dcb90c9d917d81884809b2b03bbf50aa44d84021888fca492eb4bfbc7b32"
Sep 18 23:15:00.770024 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:00.770004    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:15:00.770460 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:00.770299    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:15:01.775523 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:01.775496    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/18.log"
Sep 18 23:15:03.613433 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:03.613404    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:15:03.613798 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:03.613601    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:15:04.179283 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:04.179238    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:15:04.179468 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:04.179316    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:15:04.179665 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:04.179648    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:15:04.179850 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:04.179831    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:15:14.614139 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:14.614109    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:15:14.614647 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:14.614335    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:15:16.613708 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:16.613678    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:15:16.614094 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:16.613872    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:15:18.349384 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:18.349337    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:18.445551 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:18.445469    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:22.062489 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:22.062459    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/14.log"
Sep 18 23:15:22.063221 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:22.063204    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/18.log"
Sep 18 23:15:22.068088 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:22.068067    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:15:22.079407 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:22.079388    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/14.log"
Sep 18 23:15:22.080028 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:22.080009    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/18.log"
Sep 18 23:15:22.084997 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:22.084980    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:15:23.698261 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:23.698224    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:23.887756 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:23.887722    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:24.655867 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:24.655828    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:27.613442 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:27.613409    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:15:27.613830 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:27.613655    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:15:30.509972 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:30.509937    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:30.586936 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:30.586903    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:30.613661 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:30.613640    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:15:30.613852 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:30.613835    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:15:30.989321 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:30.989293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:31.083772 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:31.083735    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:41.616372 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:41.616340    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:15:41.616808 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:41.616542    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:15:42.614374 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:42.614342    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:15:42.614577 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:42.614557    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:15:43.836991 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:43.836952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:44.088596 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:44.088501    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:44.807982 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:44.807944    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:44.870202 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:44.870170    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:15:53.613650 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:53.613614    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:15:53.614109 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:53.613856    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:15:55.613655 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:55.613626    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:15:55.614038 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:15:55.613849    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:15:59.537036 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:15:59.537003    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:16:08.613461 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:08.613433    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:16:08.613839 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:16:08.613622    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:16:10.613324 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:10.613296    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:16:11.112057 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:11.112033    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/15.log"
Sep 18 23:16:11.112491 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:11.112475    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/14.log"
Sep 18 23:16:11.112566 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:11.112511    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed" exitCode=1
Sep 18 23:16:11.112605 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:11.112574    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"}
Sep 18 23:16:11.112648 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:11.112624    2564 scope.go:117] "RemoveContainer" containerID="2560a8ec7252a2540c404f47c33e11498b0d51fac2d6ebf4e59fa4d7d61d7149"
Sep 18 23:16:11.112924 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:11.112906    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:16:11.113128 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:16:11.113109    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:16:11.727638 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:11.727601    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:16:11.728037 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:11.727664    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:16:12.118506 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:12.118480    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/15.log"
Sep 18 23:16:12.118803 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:12.118787    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:16:12.118988 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:16:12.118968    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:16:20.613605 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:20.613577    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:16:20.613991 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:16:20.613767    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:16:21.583602 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:21.583565    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:16:21.675067 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:21.675035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:16:24.570176 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:24.570143    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:16:26.613923 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:26.613896    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:16:26.614324 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:16:26.614092    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:16:33.614090 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:33.614060    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:16:33.614574 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:16:33.614237    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:16:39.613526 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:39.613493    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:16:39.613901 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:16:39.613671    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:16:45.472615 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:45.472577    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:16:45.613473 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:45.613443    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:16:45.613684 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:16:45.613660    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:16:51.620345 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:51.620316    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:16:51.620851 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:16:51.620523    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:16:55.504669 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:16:55.504635    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:17:00.613652 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:00.613622    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:17:00.614216 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:00.613796    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:17:02.613894 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:02.613866    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:17:02.614290 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:02.614087    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:17:10.269490 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:10.269453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:17:15.613427 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:15.613352    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:17:15.613768 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:15.613554    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:17:16.710560 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:16.710525    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:17:17.613626 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:17.613595    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:17:17.613856 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:17.613836    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:17:22.400186 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:22.400146    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:17:22.455891 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:22.455849    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:17:28.614158 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:28.614120    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:17:28.614592 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:28.614393    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:17:32.614247 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:32.614214    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:17:32.614753 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:32.614477    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:17:41.617378 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:41.617350    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:17:41.617783 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:41.617551    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:17:45.613857 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:45.613817    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:17:45.614281 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:45.614071    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:17:56.614033 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:56.614001    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:17:56.614479 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:56.614196    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:17:57.613837 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:57.613805    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:17:57.614031 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:17:57.614010    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:17:59.308965 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:17:59.308924    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:18:07.614333 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:18:07.614299    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:18:07.614918 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:18:07.614509    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:18:12.613760 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:18:12.613728    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:18:12.614183 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:18:12.613914    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:18:22.613331 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:18:22.613302    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:18:22.615713 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:18:22.613514    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:18:26.613855 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:18:26.613825    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:18:26.614238 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:18:26.614052    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:18:35.614459 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:18:35.614428    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:18:35.614862 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:18:35.614614    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:18:40.613807 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:18:40.613777    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:18:40.614361 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:18:40.614005    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:18:48.613590 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:18:48.613516    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:18:48.613942 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:18:48.613718    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:18:52.613189 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:18:52.613157    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:18:52.613576 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:18:52.613381    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:19:03.614199 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:03.614166    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:19:03.614598 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:03.614416    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:19:06.613710 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:06.613682    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:19:06.614102 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:06.613873    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:19:18.614233 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:18.614204    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:19:18.614695 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:18.614411    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:19:19.613597 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:19.613566    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:19:19.613770 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:19.613750    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:19:29.613619 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:29.613585    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:19:29.614097 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:29.613788    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:19:30.613724 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:30.613694    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:19:30.614089 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:30.613903    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:19:40.613635 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:40.613606    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:19:40.614045 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:40.613793    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:19:43.613281 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:43.613240    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:19:43.613658 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:43.613458    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:19:53.613974 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:53.613936    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:19:53.614541 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:53.614174    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:19:55.620582 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:19:55.620554    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:19:55.620956 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:19:55.620722    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:20:05.613818 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:05.613786    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:20:06.245405 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:06.245379    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/19.log"
Sep 18 23:20:06.245834 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:06.245813    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/18.log"
Sep 18 23:20:06.245957 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:06.245852    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b" exitCode=1
Sep 18 23:20:06.245957 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:06.245928    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"}
Sep 18 23:20:06.246031 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:06.245991    2564 scope.go:117] "RemoveContainer" containerID="2f9f7f9fece18522ebe7a6df6921cf0a5e54cc86d1346fe6d3832959d75417c0"
Sep 18 23:20:06.246355 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:06.246335    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:20:06.246595 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:20:06.246576    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:20:07.252530 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:07.252498    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/19.log"
Sep 18 23:20:07.613433 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:07.613404    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:20:07.613641 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:20:07.613618    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:20:14.179316 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:14.179223    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:20:14.179316 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:14.179304    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:20:14.179747 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:14.179608    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:20:14.179807 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:20:14.179781    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:20:20.613606 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:20.613577    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:20:20.614062 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:20:20.613817    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:20:22.095805 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:22.095772    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/15.log"
Sep 18 23:20:22.096553 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:22.096537    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/19.log"
Sep 18 23:20:22.101431 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:22.101413    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:20:22.112699 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:22.112676    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/15.log"
Sep 18 23:20:22.113454 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:22.113437    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/19.log"
Sep 18 23:20:22.118256 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:22.118237    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:20:28.614063 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:28.614035    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:20:28.614535 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:20:28.614221    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:20:34.613750 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:34.613714    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:20:34.614198 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:20:34.613940    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:20:40.613431 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:40.613394    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:20:40.613881 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:20:40.613617    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:20:48.613560 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:48.613524    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:20:48.614034 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:20:48.613765    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:20:54.613729 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:20:54.613701    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:20:54.614094 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:20:54.613884    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:21:00.613261 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:00.613232    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:21:00.613725 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:00.613448    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:21:09.613281 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:09.613250    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:21:09.613656 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:09.613469    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:21:15.619579 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:15.619548    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:21:16.578622 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:16.578595    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/16.log"
Sep 18 23:21:16.579002 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:16.578988    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/15.log"
Sep 18 23:21:16.579058 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:16.579019    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233" exitCode=1
Sep 18 23:21:16.579058 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:16.579052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"}
Sep 18 23:21:16.579126 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:16.579098    2564 scope.go:117] "RemoveContainer" containerID="f1788e403ca0ea9527e8517cf86c9497f8e259faa0048a40a6095d72b3f0d1ed"
Sep 18 23:21:16.579435 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:16.579411    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:21:16.579628 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:16.579609    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:21:17.585680 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:17.585653    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/16.log"
Sep 18 23:21:21.728109 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:21.728072    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:21:21.728521 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:21.728136    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:21:21.728521 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:21.728506    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:21:21.728723 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:21.728693    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:21:24.613916 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:24.613882    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:21:24.614385 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:24.614134    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:21:32.613646 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:32.613618    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:21:32.614104 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:32.613810    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:21:39.613495 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:39.613462    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:21:39.613877 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:39.613652    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:21:43.613959 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:43.613927    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:21:43.614354 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:43.614107    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:21:50.613692 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:50.613661    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:21:50.614068 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:50.613865    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:21:56.615386 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:21:56.615350    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:21:56.615889 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:21:56.615644    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:22:02.614358 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:22:02.614330    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:22:02.614726 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:22:02.614519    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:22:10.613357 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:22:10.613322    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:22:10.613818 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:22:10.613511    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:22:17.613999 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:22:17.613969    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:22:17.614490 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:22:17.614160    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:22:24.613604 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:22:24.613574    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:22:24.613971 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:22:24.613783    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:22:29.613428 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:22:29.613398    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:22:29.613821 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:22:29.613586    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:22:37.613530 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:22:37.613502    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:22:37.613969 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:22:37.613740    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:22:42.614297 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:22:42.614253    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:22:42.614735 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:22:42.614455    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:22:52.614112 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:22:52.614083    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:22:52.614498 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:22:52.614282    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:22:55.613598 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:22:55.613569    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:22:55.613970 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:22:55.613751    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:23:06.613767 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:06.613735    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:23:06.614236 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:06.613989    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:23:07.613299 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:07.613254    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:23:07.613480 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:07.613460    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:23:21.616155 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:21.616063    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:23:21.616155 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:21.616133    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:23:21.616787 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:21.616330    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:23:21.616787 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:21.616380    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:23:32.613798 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:32.613769    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:23:32.616169 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:32.613948    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:23:34.613707 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:34.613679    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:23:34.614065 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:34.613848    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:23:45.614043 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:45.614009    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:23:45.614561 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:45.614183    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:23:47.613807 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:47.613775    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:23:47.614201 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:47.614027    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:23:58.614131 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:58.614100    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:23:58.614584 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:23:58.614199    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:23:58.614584 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:58.614291    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:23:58.614584 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:23:58.614382    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:24:11.616360 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:24:11.616331    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:24:11.616744 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:24:11.616557    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:24:13.613439 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:24:13.613412    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:24:13.613876 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:24:13.613620    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:24:24.613207 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:24:24.613177    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:24:24.613611 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:24:24.613379    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:24:27.614227 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:24:27.614191    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:24:27.614741 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:24:27.614468    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:24:36.511096 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:24:36.511052    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:24:37.613639 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:24:37.613611    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:24:37.614098 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:24:37.613811    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:24:40.614046 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:24:40.614018    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:24:40.614446 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:24:40.614199    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:24:50.614243 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:24:50.614170    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:24:50.614607 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:24:50.614392    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:24:52.613874 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:24:52.613844    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:24:52.614243 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:24:52.614055    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:25:01.615909 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:01.615879    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:25:01.616358 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:01.616085    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:25:03.613348 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:03.613313    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:25:03.613828 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:03.613526    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:25:12.613246 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:12.613211    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:25:12.711353 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:12.711327    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/19.log"
Sep 18 23:25:12.711471 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:12.711424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerStarted","Data":"48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"}
Sep 18 23:25:12.711662 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:12.711645    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:25:12.739694 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:12.739650    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podStartSLOduration=4669.739636162 podStartE2EDuration="1h17m49.739636162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:25:12.737733802 +0000 UTC m=+5691.594389453" watchObservedRunningTime="2026-09-18 23:25:12.739636162 +0000 UTC m=+5691.596291813"
Sep 18 23:25:13.717669 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:13.717638    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/20.log"
Sep 18 23:25:13.718135 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:13.718117    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/19.log"
Sep 18 23:25:13.718187 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:13.718159    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b" exitCode=1
Sep 18 23:25:13.718293 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:13.718255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"}
Sep 18 23:25:13.718346 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:13.718334    2564 scope.go:117] "RemoveContainer" containerID="37893d331eeedd27b17ae7c22b196a63369e2cab95842f6efe1f52e774e7923b"
Sep 18 23:25:13.718590 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:13.718575    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:25:13.718815 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:13.718796    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:25:14.179431 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:14.179403    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:25:14.724776 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:14.724745    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/20.log"
Sep 18 23:25:14.725163 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:14.725113    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:25:14.725362 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:14.725340    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:25:15.613253 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:15.613220    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:25:15.613466 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:15.613423    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:25:15.729096 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:15.729070    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:25:15.729471 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:15.729230    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:25:22.128369 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:22.128338    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/16.log"
Sep 18 23:25:22.129104 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:22.129089    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/20.log"
Sep 18 23:25:22.134010 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:22.133988    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:25:22.147074 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:22.147051    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/16.log"
Sep 18 23:25:22.147792 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:22.147776    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/20.log"
Sep 18 23:25:22.152223 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:22.152205    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:25:29.619994 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:29.619965    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:25:29.620535 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:29.620141    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:25:29.620535 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:29.620160    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:25:29.620535 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:29.620373    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:25:40.613403 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:40.613374    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:25:40.613849 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:40.613564    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:25:43.613760 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:43.613733    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:25:43.614298 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:43.613971    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:25:53.613827 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:53.613793    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:25:53.614214 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:53.614045    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:25:56.613449 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:25:56.613423    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:25:56.613824 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:25:56.613623    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:26:05.613239 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:05.613212    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:26:05.613674 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:05.613431    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:26:09.613836 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:09.613802    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:26:09.614220 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:09.614055    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:26:20.613579 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:20.613500    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:26:20.614041 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:20.613690    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:26:22.613217 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:22.613191    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:26:23.056064 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:23.056035    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/17.log"
Sep 18 23:26:23.056505 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:23.056487    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/16.log"
Sep 18 23:26:23.056631 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:23.056527    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0" exitCode=1
Sep 18 23:26:23.056631 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:23.056601    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"}
Sep 18 23:26:23.056734 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:23.056664    2564 scope.go:117] "RemoveContainer" containerID="58b23a70e5fefd61a17ffae271d09b3761d2e3b7d6c516c65caf2fa8ba773233"
Sep 18 23:26:23.057002 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:23.056983    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:26:23.057251 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:23.057232    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:26:24.063058 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:24.063027    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/17.log"
Sep 18 23:26:31.727326 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:31.727282    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:26:31.727326 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:31.727343    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:26:31.727823 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:31.727597    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:26:31.727823 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:31.727787    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:26:32.614159 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:32.614131    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:26:32.614351 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:32.614330    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:26:44.613250 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:44.613213    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:26:44.613738 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:44.613473    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:26:45.613875 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:45.613844    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:26:45.614369 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:45.614037    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:26:56.613710 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:56.613680    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:26:56.614144 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:56.613861    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:26:59.613337 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:26:59.613308    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:26:59.613738 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:26:59.613494    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:27:07.614188 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:07.614157    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:27:07.614653 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:07.614404    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:27:12.613877 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:12.613846    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:27:12.614242 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:12.614047    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:27:14.770533 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:14.770503    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-prefill-7f8db478bb-9l47n_01e70533-b9ac-4b8d-a073-838c050337d9/storage-initializer/0.log"
Sep 18 23:27:14.778833 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:14.778812    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-prefill-7f8db478bb-9l47n_01e70533-b9ac-4b8d-a073-838c050337d9/main/0.log"
Sep 18 23:27:16.026026 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:16.025992    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"]
Sep 18 23:27:16.052150 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:16.052116    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:27:16.111877 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:16.111847    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:27:21.617063 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:21.617035    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:27:21.617526 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:21.617231    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:27:23.613803 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:23.613774    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:27:23.614283 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:23.613933    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:27:31.050918 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.050870    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main" containerID="cri-o://fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda" gracePeriod=45
Sep 18 23:27:31.805947 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.805926    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:27:31.879548 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.879467    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-dshm\") pod \"01e70533-b9ac-4b8d-a073-838c050337d9\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") "
Sep 18 23:27:31.879548 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.879541    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e70533-b9ac-4b8d-a073-838c050337d9-tls-certs\") pod \"01e70533-b9ac-4b8d-a073-838c050337d9\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") "
Sep 18 23:27:31.879773 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.879567    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-tmp-dir\") pod \"01e70533-b9ac-4b8d-a073-838c050337d9\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") "
Sep 18 23:27:31.879773 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.879611    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-model-cache\") pod \"01e70533-b9ac-4b8d-a073-838c050337d9\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") "
Sep 18 23:27:31.879773 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.879637    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8kq8\" (UniqueName: \"kubernetes.io/projected/01e70533-b9ac-4b8d-a073-838c050337d9-kube-api-access-z8kq8\") pod \"01e70533-b9ac-4b8d-a073-838c050337d9\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") "
Sep 18 23:27:31.879773 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.879663    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-home\") pod \"01e70533-b9ac-4b8d-a073-838c050337d9\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") "
Sep 18 23:27:31.879773 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.879688    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-kserve-provision-location\") pod \"01e70533-b9ac-4b8d-a073-838c050337d9\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") "
Sep 18 23:27:31.879773 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.879719    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e70533-b9ac-4b8d-a073-838c050337d9-cabundle-cert\") pod \"01e70533-b9ac-4b8d-a073-838c050337d9\" (UID: \"01e70533-b9ac-4b8d-a073-838c050337d9\") "
Sep 18 23:27:31.880073 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.879831    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "01e70533-b9ac-4b8d-a073-838c050337d9" (UID: "01e70533-b9ac-4b8d-a073-838c050337d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:31.880129 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.880088    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:27:31.880184 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.880148    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e70533-b9ac-4b8d-a073-838c050337d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01e70533-b9ac-4b8d-a073-838c050337d9" (UID: "01e70533-b9ac-4b8d-a073-838c050337d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:27:31.880462 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.880427    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-home" (OuterVolumeSpecName: "home") pod "01e70533-b9ac-4b8d-a073-838c050337d9" (UID: "01e70533-b9ac-4b8d-a073-838c050337d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:31.882021 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.881989    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e70533-b9ac-4b8d-a073-838c050337d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01e70533-b9ac-4b8d-a073-838c050337d9" (UID: "01e70533-b9ac-4b8d-a073-838c050337d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:27:31.882321 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.882300    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-dshm" (OuterVolumeSpecName: "dshm") pod "01e70533-b9ac-4b8d-a073-838c050337d9" (UID: "01e70533-b9ac-4b8d-a073-838c050337d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:31.882607 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.882572    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e70533-b9ac-4b8d-a073-838c050337d9-kube-api-access-z8kq8" (OuterVolumeSpecName: "kube-api-access-z8kq8") pod "01e70533-b9ac-4b8d-a073-838c050337d9" (UID: "01e70533-b9ac-4b8d-a073-838c050337d9"). InnerVolumeSpecName "kube-api-access-z8kq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:27:31.892168 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.892141    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01e70533-b9ac-4b8d-a073-838c050337d9" (UID: "01e70533-b9ac-4b8d-a073-838c050337d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:31.905873 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.905850    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01e70533-b9ac-4b8d-a073-838c050337d9" (UID: "01e70533-b9ac-4b8d-a073-838c050337d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:31.981067 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.981048    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e70533-b9ac-4b8d-a073-838c050337d9-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:27:31.981067 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.981068    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:27:31.981189 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.981078    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e70533-b9ac-4b8d-a073-838c050337d9-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:27:31.981189 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.981085    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:27:31.981189 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.981093    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8kq8\" (UniqueName: \"kubernetes.io/projected/01e70533-b9ac-4b8d-a073-838c050337d9-kube-api-access-z8kq8\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:27:31.981189 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.981101    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:27:31.981189 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:31.981109    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e70533-b9ac-4b8d-a073-838c050337d9-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:27:32.390904 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.390867    2564 generic.go:356] "Generic (PLEG): container finished" podID="01e70533-b9ac-4b8d-a073-838c050337d9" containerID="fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda" exitCode=0
Sep 18 23:27:32.391391 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.390935    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"
Sep 18 23:27:32.391391 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.390942    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" event={"ID":"01e70533-b9ac-4b8d-a073-838c050337d9","Type":"ContainerDied","Data":"fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda"}
Sep 18 23:27:32.391391 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.391014    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n" event={"ID":"01e70533-b9ac-4b8d-a073-838c050337d9","Type":"ContainerDied","Data":"f1b50aed842fd9c294b11b34e44298f74a74c1989ff5ed105fc90391c6cc05be"}
Sep 18 23:27:32.391391 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.391037    2564 scope.go:117] "RemoveContainer" containerID="fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda"
Sep 18 23:27:32.402667 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.402367    2564 scope.go:117] "RemoveContainer" containerID="9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be"
Sep 18 23:27:32.410475 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.410457    2564 scope.go:117] "RemoveContainer" containerID="fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda"
Sep 18 23:27:32.410724 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:32.410704    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda\": container with ID starting with fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda not found: ID does not exist" containerID="fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda"
Sep 18 23:27:32.410792 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.410732    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda"} err="failed to get container status \"fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda\": rpc error: code = NotFound desc = could not find container \"fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda\": container with ID starting with fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda not found: ID does not exist"
Sep 18 23:27:32.410792 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.410748    2564 scope.go:117] "RemoveContainer" containerID="9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be"
Sep 18 23:27:32.410980 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:32.410961    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be\": container with ID starting with 9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be not found: ID does not exist" containerID="9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be"
Sep 18 23:27:32.411028 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.410987    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be"} err="failed to get container status \"9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be\": rpc error: code = NotFound desc = could not find container \"9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be\": container with ID starting with 9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be not found: ID does not exist"
Sep 18 23:27:32.414192 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.414173    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"]
Sep 18 23:27:32.417739 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:32.417717    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n"]
Sep 18 23:27:33.619833 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:33.619799    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" path="/var/lib/kubelet/pods/01e70533-b9ac-4b8d-a073-838c050337d9/volumes"
Sep 18 23:27:36.614107 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:36.614078    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:27:36.614540 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:36.614202    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:27:36.614540 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:36.614302    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:27:36.614540 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:36.614380    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:27:51.615977 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:51.615892    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:27:51.615977 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:27:51.615946    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:27:51.616500 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:51.616112    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:27:51.616500 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:27:51.616145    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:28:06.033951 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:06.033904    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:28:06.134155 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:06.134124    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:28:06.613636 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:06.613611    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:28:06.613800 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:06.613719    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:28:06.613800 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:06.613774    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:28:06.613888 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:06.613850    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:28:18.614374 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:18.614344    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:28:18.614748 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:18.614511    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:28:19.613557 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:19.613526    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:28:19.613784 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:19.613748    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:28:31.442541 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:31.442501    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:28:31.617146 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:31.617120    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:28:31.617442 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:31.617407    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:28:31.654944 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:31.654919    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:28:33.613286 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:33.613244    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:28:33.615603 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:33.613453    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:28:43.613779 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:43.613749    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:28:43.614320 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:43.613970    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:28:45.613502 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:45.613475    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:28:45.613904 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:45.613640    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:28:48.940634 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:48.940597    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:28:54.613203 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:54.613177    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:28:54.613606 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:54.613419    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:28:57.613311 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:28:57.613257    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:28:57.613702 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:28:57.613440    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:29:09.613305 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:29:09.613258    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:29:09.613679 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:29:09.613489    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:29:12.613285 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:29:12.613239    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:29:12.613677 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:29:12.613451    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:29:21.615842 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:29:21.615755    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:29:21.616446 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:29:21.615949    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:29:23.613625 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:29:23.613595    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:29:23.614005 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:29:23.613795    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:29:34.613393 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:29:34.613360    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:29:34.613877 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:29:34.613466    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:29:34.613877 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:29:34.613546    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:29:34.613877 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:29:34.613667    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:29:45.614366 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:29:45.614335    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:29:45.614729 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:29:45.614495    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:29:46.613883 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:29:46.613850    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:29:46.614102 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:29:46.614076    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:29:58.613522 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:29:58.613489    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:29:58.614046 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:29:58.613730    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:30:00.613926 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:00.613897    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:30:00.614309 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:00.614069    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:30:12.613260 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:12.613220    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:30:12.613665 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:12.613473    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:30:12.613665 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:12.613508    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:30:12.613750 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:12.613701    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:30:22.161497 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:22.161469    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/17.log"
Sep 18 23:30:22.162224 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:22.162204    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/20.log"
Sep 18 23:30:22.166994 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:22.166975    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:30:22.178148 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:22.178125    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/17.log"
Sep 18 23:30:22.178867 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:22.178850    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/20.log"
Sep 18 23:30:22.183585 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:22.183569    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:30:23.613886 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:23.613855    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:30:23.614282 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:23.614080    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:30:27.613599 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:27.613568    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:30:28.200695 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:28.200666    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/21.log"
Sep 18 23:30:28.201112 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:28.201096    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/20.log"
Sep 18 23:30:28.201167 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:28.201132    2564 generic.go:356] "Generic (PLEG): container finished" podID="cea0a11c-8723-4ee5-b1c7-974bb48f243b" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306" exitCode=1
Sep 18 23:30:28.201213 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:28.201163    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" event={"ID":"cea0a11c-8723-4ee5-b1c7-974bb48f243b","Type":"ContainerDied","Data":"fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"}
Sep 18 23:30:28.201254 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:28.201216    2564 scope.go:117] "RemoveContainer" containerID="48060153766b80b9fec30aac6a4d35fab4f5ba0d446449139a038eb7777d4d3b"
Sep 18 23:30:28.201619 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:28.201589    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:30:28.201797 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:28.201775    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:30:29.207636 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:29.207607    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-flow-control-smoke-1e98f46d_fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_cea0a11c-8723-4ee5-b1c7-974bb48f243b/main/21.log"
Sep 18 23:30:32.712165 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:32.712132    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:30:32.712685 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:32.712447    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:30:32.712685 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:32.712605    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:30:34.179613 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:34.179579    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk"
Sep 18 23:30:34.179988 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:34.179963    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:30:34.180145 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:34.180126    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:30:36.716848 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.716814    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"]
Sep 18 23:30:36.717218 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.717168    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="storage-initializer"
Sep 18 23:30:36.717218 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.717179    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="storage-initializer"
Sep 18 23:30:36.717218 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.717211    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main"
Sep 18 23:30:36.717218 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.717217    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main"
Sep 18 23:30:36.717392 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.717292    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e70533-b9ac-4b8d-a073-838c050337d9" containerName="main"
Sep 18 23:30:36.721577 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.721559    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.725020 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.724990    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-zwj7d\""
Sep 18 23:30:36.725244 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.725228    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:30:36.725364 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.725255    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 23:30:36.725364 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.725251    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 23:30:36.725364 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.725307    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 23:30:36.725364 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.725315    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 23:30:36.735171 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.735148    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"]
Sep 18 23:30:36.804638 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.804601    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kserve-provision-location\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.804825 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.804644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-cabundle-cert\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.804825 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.804704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59r8d\" (UniqueName: \"kubernetes.io/projected/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kube-api-access-59r8d\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.804825 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.804755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tls-certs\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.804825 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.804780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tmp-dir\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.804825 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.804801    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-home\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.805032 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.804838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-model-cache\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.805032 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.804863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-dshm\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.905623 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.905592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kserve-provision-location\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.905623 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.905627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-cabundle-cert\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.905854 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.905654    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59r8d\" (UniqueName: \"kubernetes.io/projected/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kube-api-access-59r8d\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.905854 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.905688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tls-certs\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.905854 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.905707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tmp-dir\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.905854 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.905721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-home\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.905854 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.905740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-model-cache\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.905854 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.905756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-dshm\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.906172 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.906023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kserve-provision-location\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.906172 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.906123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-home\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.906172 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.906162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tmp-dir\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.906353 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.906331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-model-cache\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.906447 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.906433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-cabundle-cert\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.907918 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.907896    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-dshm\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.908330 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.908311    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tls-certs\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:36.914080 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:36.914058    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59r8d\" (UniqueName: \"kubernetes.io/projected/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kube-api-access-59r8d\") pod \"prestop-hook-test-kserve-767b458bf9-qgpml\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:37.032520 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:37.032495    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:37.309247 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:37.309224    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"]
Sep 18 23:30:37.311336 ip-10-0-130-79 kubenswrapper[2564]: W0918 23:30:37.311308    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff74a1fc_f290_43ee_9ad6_6fea6ed4c250.slice/crio-3848f98302032979a857329cadd9195f05280d86e36692cbba8560cd9be666fa WatchSource:0}: Error finding container 3848f98302032979a857329cadd9195f05280d86e36692cbba8560cd9be666fa: Status 404 returned error can't find the container with id 3848f98302032979a857329cadd9195f05280d86e36692cbba8560cd9be666fa
Sep 18 23:30:37.614131 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:37.614050    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:30:37.614326 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:37.614299    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:30:38.252090 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:38.252049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" event={"ID":"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250","Type":"ContainerStarted","Data":"7099fb2a8cdf933f28cee976e5f01f1b5c1482b87e7b7b294dd5fde76794caed"}
Sep 18 23:30:38.252596 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:38.252111    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" event={"ID":"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250","Type":"ContainerStarted","Data":"3848f98302032979a857329cadd9195f05280d86e36692cbba8560cd9be666fa"}
Sep 18 23:30:40.264018 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:40.263980    2564 generic.go:356] "Generic (PLEG): container finished" podID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerID="7099fb2a8cdf933f28cee976e5f01f1b5c1482b87e7b7b294dd5fde76794caed" exitCode=0
Sep 18 23:30:40.264437 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:40.264054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" event={"ID":"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250","Type":"ContainerDied","Data":"7099fb2a8cdf933f28cee976e5f01f1b5c1482b87e7b7b294dd5fde76794caed"}
Sep 18 23:30:41.270747 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:41.270712    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" event={"ID":"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250","Type":"ContainerStarted","Data":"8d626eb3fd1d804dbc72feeeed737b0c42ef7c2e17ed21f13810207699aa6fed"}
Sep 18 23:30:41.295397 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:41.295349    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" podStartSLOduration=3.154247441 podStartE2EDuration="5.295333691s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.14108625s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:30:41.292950246 +0000 UTC m=+6020.149605897" watchObservedRunningTime="2026-09-18 23:30:41.295333691 +0000 UTC m=+6020.151989343"
Sep 18 23:30:42.032589 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:42.032558    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:47.033549 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:47.033462    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:30:47.034764 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:47.034733    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" podUID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 23:30:47.614383 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:47.614349    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:30:47.614609 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:47.614582    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:30:52.613872 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:52.613841    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:30:52.614224 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:30:52.614032    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:30:57.033438 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:57.033396    2564 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" podUID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 23:30:58.141202 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:58.141133    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"]
Sep 18 23:30:58.175111 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:58.175075    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:30:59.526503 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:30:59.526470    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:31:02.613591 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:02.613560    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:31:02.613981 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:31:02.613803    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:31:06.614361 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:06.614331    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:31:06.614806 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:31:06.614560    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:31:13.166920 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:13.166873    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" podUID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerName="main" containerID="cri-o://8d626eb3fd1d804dbc72feeeed737b0c42ef7c2e17ed21f13810207699aa6fed" gracePeriod=45
Sep 18 23:31:15.614209 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:15.614172    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:31:15.614691 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:31:15.614455    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:31:20.613258 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:20.613230    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:31:20.613701 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:31:20.613461    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:31:28.613411 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:28.613379    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:31:28.613799 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:31:28.613590    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:31:31.617448 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:31.617416    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:31:32.531565 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:32.531537    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/17.log"
Sep 18 23:31:32.531758 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:32.531651    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerStarted","Data":"7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"}
Sep 18 23:31:32.531976 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:32.531953    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:31:32.551443 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:32.551391    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podStartSLOduration=4061.551371595 podStartE2EDuration="1h7m41.551371595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:31:32.550925156 +0000 UTC m=+6071.407580808" watchObservedRunningTime="2026-09-18 23:31:32.551371595 +0000 UTC m=+6071.408027248"
Sep 18 23:31:40.613965 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:40.613936    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:31:40.614438 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:31:40.614146    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:31:41.728977 ip-10-0-130-79 kubenswrapper[2564]: W0918 23:31:41.728949    2564 logging.go:55] [core] [Channel #227 SubChannel #228]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.35:9003", ServerName: "10.132.0.35:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.35:9003: connect: connection refused"
Sep 18 23:31:42.728435 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:42.728388    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.35:9003\" within 1s: context deadline exceeded"
Sep 18 23:31:47.860852 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:47.860783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:31:48.418522 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:48.418482    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:31:50.134191 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:50.134156    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:31:51.728164 ip-10-0-130-79 kubenswrapper[2564]: W0918 23:31:51.728138    2564 logging.go:55] [core] [Channel #229 SubChannel #230]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.35:9003", ServerName: "10.132.0.35:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.35:9003: connect: connection refused"
Sep 18 23:31:52.613736 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:52.613700    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:31:52.613949 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:31:52.613925    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:31:52.638888 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:52.638866    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/18.log"
Sep 18 23:31:52.639380 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:52.639362    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/17.log"
Sep 18 23:31:52.639458 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:52.639398    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7" exitCode=1
Sep 18 23:31:52.639458 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:52.639452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" event={"ID":"9b30dbc3-fb8a-439e-98c4-73c6cd5f7864","Type":"ContainerDied","Data":"7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"}
Sep 18 23:31:52.639537 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:52.639495    2564 scope.go:117] "RemoveContainer" containerID="4f4c65a4f3af121472ebf6cfd92f3cd2a303856d919a9bc77d73cc5f238438e0"
Sep 18 23:31:52.639827 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:52.639807    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:31:52.640048 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:31:52.640021    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:31:52.728030 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:52.727998    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.35:9003\" within 1s: context deadline exceeded"
Sep 18 23:31:53.650302 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:53.650246    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_9b30dbc3-fb8a-439e-98c4-73c6cd5f7864/main/18.log"
Sep 18 23:31:58.675351 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.675326    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-767b458bf9-qgpml_ff74a1fc-f290-43ee-9ad6-6fea6ed4c250/main/0.log"
Sep 18 23:31:58.675774 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.675738    2564 generic.go:356] "Generic (PLEG): container finished" podID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerID="8d626eb3fd1d804dbc72feeeed737b0c42ef7c2e17ed21f13810207699aa6fed" exitCode=137
Sep 18 23:31:58.675833 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.675812    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" event={"ID":"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250","Type":"ContainerDied","Data":"8d626eb3fd1d804dbc72feeeed737b0c42ef7c2e17ed21f13810207699aa6fed"}
Sep 18 23:31:58.813421 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.813398    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-767b458bf9-qgpml_ff74a1fc-f290-43ee-9ad6-6fea6ed4c250/main/0.log"
Sep 18 23:31:58.816021 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.816005    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:31:58.885407 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.885374    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-home\") pod \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") "
Sep 18 23:31:58.885573 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.885416    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tls-certs\") pod \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") "
Sep 18 23:31:58.885573 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.885489    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-dshm\") pod \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") "
Sep 18 23:31:58.885573 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.885523    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59r8d\" (UniqueName: \"kubernetes.io/projected/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kube-api-access-59r8d\") pod \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") "
Sep 18 23:31:58.885739 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.885592    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tmp-dir\") pod \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") "
Sep 18 23:31:58.885739 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.885623    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-cabundle-cert\") pod \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") "
Sep 18 23:31:58.885739 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.885667    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kserve-provision-location\") pod \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") "
Sep 18 23:31:58.885739 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.885702    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-model-cache\") pod \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\" (UID: \"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250\") "
Sep 18 23:31:58.885930 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.885771    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-home" (OuterVolumeSpecName: "home") pod "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" (UID: "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:31:58.886020 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.886004    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:31:58.886329 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.886302    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" (UID: "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:31:58.886450 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.886017    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" (UID: "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:31:58.886546 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.886526    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" (UID: "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:31:58.887874 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.887847    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kube-api-access-59r8d" (OuterVolumeSpecName: "kube-api-access-59r8d") pod "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" (UID: "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250"). InnerVolumeSpecName "kube-api-access-59r8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:31:58.888059 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.888038    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-dshm" (OuterVolumeSpecName: "dshm") pod "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" (UID: "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:31:58.888128 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.888047    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" (UID: "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:31:58.909807 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.909781    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" (UID: "ff74a1fc-f290-43ee-9ad6-6fea6ed4c250"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:31:58.987182 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.987116    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:31:58.987182 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.987140    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59r8d\" (UniqueName: \"kubernetes.io/projected/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kube-api-access-59r8d\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:31:58.987182 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.987153    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:31:58.987182 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.987165    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:31:58.987182 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.987177    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:31:58.987436 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.987189    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:31:58.987436 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:58.987200    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 23:31:59.681469 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:59.681446    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-767b458bf9-qgpml_ff74a1fc-f290-43ee-9ad6-6fea6ed4c250/main/0.log"
Sep 18 23:31:59.681855 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:59.681834    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml" event={"ID":"ff74a1fc-f290-43ee-9ad6-6fea6ed4c250","Type":"ContainerDied","Data":"3848f98302032979a857329cadd9195f05280d86e36692cbba8560cd9be666fa"}
Sep 18 23:31:59.681926 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:59.681872    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"
Sep 18 23:31:59.681926 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:59.681885    2564 scope.go:117] "RemoveContainer" containerID="8d626eb3fd1d804dbc72feeeed737b0c42ef7c2e17ed21f13810207699aa6fed"
Sep 18 23:31:59.690431 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:59.690410    2564 scope.go:117] "RemoveContainer" containerID="7099fb2a8cdf933f28cee976e5f01f1b5c1482b87e7b7b294dd5fde76794caed"
Sep 18 23:31:59.700797 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:59.700772    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"]
Sep 18 23:31:59.712806 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:31:59.707551    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-qgpml"]
Sep 18 23:32:01.326151 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:01.326117    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:32:01.620544 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:01.620468    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" path="/var/lib/kubelet/pods/ff74a1fc-f290-43ee-9ad6-6fea6ed4c250/volumes"
Sep 18 23:32:01.727704 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:01.727675    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr"
Sep 18 23:32:01.728044 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:01.728026    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:32:01.728251 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:01.728228    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:32:05.613741 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:05.613710    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:32:05.614166 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:05.613881    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:32:14.614170 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:14.614078    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:32:14.614660 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:14.614325    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:32:19.613361 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:19.613331    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:32:19.613781 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:19.613585    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:32:25.613275 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:25.613236    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:32:25.613671 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:25.613467    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:32:30.614055 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:30.614022    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:32:30.614451 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:30.614182    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:32:37.613490 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:37.613459    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:32:37.613995 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:37.613617    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:32:44.613167 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:44.613135    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:32:44.613539 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:44.613325    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:32:51.616105 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:51.616078    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:32:51.616598 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:51.616256    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:32:56.614354 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:32:56.614320    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:32:56.614742 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:32:56.614529    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:33:05.613275 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:05.613236    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:33:05.613643 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:05.613443    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:33:07.273719 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:07.273686    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:33:10.613314 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:10.613287    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:33:10.613692 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:10.613504    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:33:10.991627 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:10.991532    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:33:11.049840 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:11.049771    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:33:13.133389 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:13.133339    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:33:13.191905 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:13.191868    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:33:13.716828 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:13.716791    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:33:19.677365 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:19.677318    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:33:19.731131 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:19.731100    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:33:20.613391 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:20.613363    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:33:20.613571 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:20.613538    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:33:23.613447 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:23.613416    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:33:23.613902 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:23.613591    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:33:31.615678 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:31.615649    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:33:31.616097 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:31.615886    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:33:33.820698 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:33.820666    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/0.log"
Sep 18 23:33:33.824123 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:33.824097    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/1.log"
Sep 18 23:33:34.609875 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:34.609848    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/0.log"
Sep 18 23:33:34.613951 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:34.613930    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/1.log"
Sep 18 23:33:35.342007 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:35.341954    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/0.log"
Sep 18 23:33:35.346093 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:35.346064    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/1.log"
Sep 18 23:33:35.613375 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:35.613300    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:33:35.613521 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:35.613501    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:33:36.065425 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:36.065396    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/0.log"
Sep 18 23:33:36.069839 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:36.069812    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/1.log"
Sep 18 23:33:37.130167 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:37.130135    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:37.156973 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:37.156947    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:38.386646 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:38.386621    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:38.414169 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:38.414143    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:39.905099 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:39.905069    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:39.922678 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:39.922652    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:41.161467 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:41.161434    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:41.180449 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:41.180417    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:42.202664 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:42.202629    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:42.220780 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:42.220756    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:43.285138 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:43.285111    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:43.308562 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:43.308539    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:44.354529 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:44.354454    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:44.370572 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:44.370541    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:44.613673 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:44.613609    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:33:44.613804 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:44.613778    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:33:45.429999 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:45.429967    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:45.446582 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:45.446557    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:46.456725 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:46.456696    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:46.472509 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:46.472484    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:47.647708 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:47.647678    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:47.682101 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:47.682074    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:48.613490 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:48.613464    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:33:48.613660 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:48.613639    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:33:48.729283 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:48.729239    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:48.745676 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:48.745647    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:49.779742 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:49.779714    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:49.795411 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:49.795389    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:50.944790 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:50.944759    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:50.963393 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:50.963368    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:51.996879 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:51.996850    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:52.013077 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:52.013055    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:53.030499 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:53.030468    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:53.047888 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:53.047858    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:54.154389 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:54.154361    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:54.171232 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:54.171198    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:55.200938 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:55.200902    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:55.223467 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:55.223432    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:56.247229 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:56.247202    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:56.262607 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:56.262579    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:56.613414 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:56.613393    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:33:56.613589 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:56.613570    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:33:57.276966 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:57.276936    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:57.294498 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:57.294473    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:58.339803 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:58.339767    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:58.355584 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:58.355562    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:59.373371 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:59.373340    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:33:59.396685 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:59.396660    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:33:59.613629 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:33:59.613605    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:33:59.613810 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:33:59.613792    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:34:00.502900 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:00.502874    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:00.521125 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:00.521102    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:01.560215 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:01.560184    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:01.576230 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:01.576206    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:02.647629 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:02.647596    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:02.668105 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:02.668082    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:03.706335 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:03.706311    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:03.721444 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:03.721406    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:04.749122 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:04.749089    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:04.766548 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:04.766522    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:05.839792 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:05.839758    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:05.854956 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:05.854932    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:06.869961 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:06.869931    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:06.888427 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:06.888404    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:07.933108 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:07.933077    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:07.956788 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:07.956767    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:09.105737 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:09.105702    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:09.122117 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:09.122081    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:10.189074 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:10.189034    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:10.207379 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:10.207347    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:11.256944 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:11.256918    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:11.275206 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:11.275181    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:11.615316 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:11.615294    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:34:11.615458 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:11.615341    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:34:11.615498 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:34:11.615472    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:34:11.615535 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:34:11.615505    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:34:12.320413 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:12.320382    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:12.336235 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:12.336212    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:13.346546 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:13.346516    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:13.366160 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:13.366127    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:14.374861 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:14.374831    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:14.389928 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:14.389899    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:15.383879 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:15.383847    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:15.398580 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:15.398551    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:16.404478 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:16.404450    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:16.419778 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:16.419744    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:18.415174 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:18.415139    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:34:18.551762 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:18.551730    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:34:18.609951 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:18.609920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:34:21.601681 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:21.601645    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5nrdf_c2f70242-dfd4-42f7-9661-c888e8ca26d4/global-pull-secret-syncer/0.log"
Sep 18 23:34:21.737616 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:21.737581    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log"
Sep 18 23:34:21.865984 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:21.865907    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-79.ec2.internal_bc3d8b9722788fbfdcb8bc7f7b68e6ec/haproxy/0.log"
Sep 18 23:34:22.613695 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:22.613666    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:34:22.614071 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:22.613758    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:34:22.614071 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:34:22.613853    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:34:22.614071 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:34:22.613950    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:34:24.535834 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:24.535799    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:34:24.591582 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:24.591555    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-485xb_b1d3209a-ee8e-41b5-81c8-e0d01fc4bdd6/konnectivity-agent/0.log" limit=1024
Sep 18 23:34:27.375707 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:27.375673    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2z8v7_0141d54d-83f6-412f-b003-6c8b30e52d7c/limitador/0.log"
Sep 18 23:34:27.406648 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:27.406619    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tbv6j_9c07c347-e82c-4dc2-b1c7-2ef727ccd7b4/manager/0.log"
Sep 18 23:34:28.682538 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:28.682507    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e7fe62f6-9711-431b-8167-ba977c1f9b75/alertmanager/0.log"
Sep 18 23:34:28.706031 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:28.706012    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e7fe62f6-9711-431b-8167-ba977c1f9b75/config-reloader/0.log"
Sep 18 23:34:28.743455 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:28.743436    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e7fe62f6-9711-431b-8167-ba977c1f9b75/kube-rbac-proxy-web/0.log"
Sep 18 23:34:28.768960 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:28.768936    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e7fe62f6-9711-431b-8167-ba977c1f9b75/kube-rbac-proxy/0.log"
Sep 18 23:34:28.791149 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:28.791123    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e7fe62f6-9711-431b-8167-ba977c1f9b75/kube-rbac-proxy-metric/0.log"
Sep 18 23:34:28.824889 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:28.824871    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e7fe62f6-9711-431b-8167-ba977c1f9b75/prom-label-proxy/0.log"
Sep 18 23:34:28.854774 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:28.854755    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e7fe62f6-9711-431b-8167-ba977c1f9b75/init-config-reloader/0.log"
Sep 18 23:34:28.925446 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:28.925422    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-pmrll_8e61b030-2e75-4215-9cd0-db32078129c1/cluster-monitoring-operator/0.log"
Sep 18 23:34:29.051914 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.051887    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cc9c9ffcf-xttz5_4c3fc516-ff87-4c33-a504-9433fe34f0a7/metrics-server/0.log"
Sep 18 23:34:29.254305 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.254251    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nv98w_f5f7c83d-684c-4118-8d5d-443f8cdb46ee/node-exporter/0.log"
Sep 18 23:34:29.286426 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.286399    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nv98w_f5f7c83d-684c-4118-8d5d-443f8cdb46ee/kube-rbac-proxy/0.log"
Sep 18 23:34:29.311135 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.311062    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nv98w_f5f7c83d-684c-4118-8d5d-443f8cdb46ee/init-textfile/0.log"
Sep 18 23:34:29.520789 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.520762    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/0.log"
Sep 18 23:34:29.537044 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.537012    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/prometheus/1.log"
Sep 18 23:34:29.557205 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.557176    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/config-reloader/0.log"
Sep 18 23:34:29.582288 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.582236    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/thanos-sidecar/0.log"
Sep 18 23:34:29.604954 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.604926    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/kube-rbac-proxy-web/0.log"
Sep 18 23:34:29.629024 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.628994    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/kube-rbac-proxy/0.log"
Sep 18 23:34:29.652195 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.652166    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/kube-rbac-proxy-thanos/0.log"
Sep 18 23:34:29.674970 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.674944    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_28a7731a-fd85-47b7-b7c9-982264658a58/init-config-reloader/0.log"
Sep 18 23:34:29.709585 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.709546    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-pk5s2_ce747ee0-3d50-49bb-9f7c-08f70793d5e1/prometheus-operator/0.log"
Sep 18 23:34:29.727886 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.727856    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-pk5s2_ce747ee0-3d50-49bb-9f7c-08f70793d5e1/kube-rbac-proxy/0.log"
Sep 18 23:34:29.753024 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.752988    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-lqv8c_865926b3-b843-49ce-bd12-591aa6b8c33a/prometheus-operator-admission-webhook/0.log"
Sep 18 23:34:29.817435 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.817353    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5644b867f7-lxt8l_5ce20b8b-f247-46b2-8319-962864241977/telemeter-client/0.log"
Sep 18 23:34:29.845989 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.845964    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5644b867f7-lxt8l_5ce20b8b-f247-46b2-8319-962864241977/reload/0.log"
Sep 18 23:34:29.869776 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:29.869746    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5644b867f7-lxt8l_5ce20b8b-f247-46b2-8319-962864241977/kube-rbac-proxy/0.log"
Sep 18 23:34:31.229506 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.229474    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-v598q_f3133e4f-8bf5-41fe-8467-ca57a8a1a309/networking-console-plugin/0.log"
Sep 18 23:34:31.445819 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.445784    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"]
Sep 18 23:34:31.446179 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.446166    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerName="main"
Sep 18 23:34:31.446224 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.446180    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerName="main"
Sep 18 23:34:31.446224 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.446194    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerName="storage-initializer"
Sep 18 23:34:31.446224 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.446200    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerName="storage-initializer"
Sep 18 23:34:31.446349 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.446292    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff74a1fc-f290-43ee-9ad6-6fea6ed4c250" containerName="main"
Sep 18 23:34:31.449740 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.449720    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.452154 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.452130    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s6w2r\"/\"default-dockercfg-hpdwh\""
Sep 18 23:34:31.452154 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.452147    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6w2r\"/\"openshift-service-ca.crt\""
Sep 18 23:34:31.452341 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.452203    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6w2r\"/\"kube-root-ca.crt\""
Sep 18 23:34:31.458692 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.458672    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"]
Sep 18 23:34:31.605312 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.605262    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-proc\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.605490 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.605330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-lib-modules\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.605490 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.605375    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8xc\" (UniqueName: \"kubernetes.io/projected/20173dd3-cebb-4834-accc-d4424a7df574-kube-api-access-nl8xc\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.605490 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.605448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-podres\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.605490 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.605474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-sys\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.706187 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.706161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8xc\" (UniqueName: \"kubernetes.io/projected/20173dd3-cebb-4834-accc-d4424a7df574-kube-api-access-nl8xc\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.706335 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.706209    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-podres\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.706335 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.706227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-sys\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.706335 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.706262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-proc\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.706335 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.706302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-lib-modules\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.706553 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.706361    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-podres\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.706553 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.706366    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-sys\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.706553 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.706370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-proc\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.706553 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.706417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20173dd3-cebb-4834-accc-d4424a7df574-lib-modules\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.715161 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.715141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8xc\" (UniqueName: \"kubernetes.io/projected/20173dd3-cebb-4834-accc-d4424a7df574-kube-api-access-nl8xc\") pod \"perf-node-gather-daemonset-6wckh\" (UID: \"20173dd3-cebb-4834-accc-d4424a7df574\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:31.760037 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:31.760017    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:32.032781 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:32.032755    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"]
Sep 18 23:34:32.034723 ip-10-0-130-79 kubenswrapper[2564]: W0918 23:34:32.034695    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod20173dd3_cebb_4834_accc_d4424a7df574.slice/crio-9a22dd808bf7b38a5e2a7cdc315726010b78f3d13e886d43b8cb3173604bc348 WatchSource:0}: Error finding container 9a22dd808bf7b38a5e2a7cdc315726010b78f3d13e886d43b8cb3173604bc348: Status 404 returned error can't find the container with id 9a22dd808bf7b38a5e2a7cdc315726010b78f3d13e886d43b8cb3173604bc348
Sep 18 23:34:32.406721 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:32.406644    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh" event={"ID":"20173dd3-cebb-4834-accc-d4424a7df574","Type":"ContainerStarted","Data":"5a45096fd6e56d667040c963143190852b18dfddebc7ebd40184e60f6062b5ff"}
Sep 18 23:34:32.406721 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:32.406700    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh" event={"ID":"20173dd3-cebb-4834-accc-d4424a7df574","Type":"ContainerStarted","Data":"9a22dd808bf7b38a5e2a7cdc315726010b78f3d13e886d43b8cb3173604bc348"}
Sep 18 23:34:32.407116 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:32.406773    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:32.426886 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:32.426837    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh" podStartSLOduration=1.426820107 podStartE2EDuration="1.426820107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 23:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:34:32.425681664 +0000 UTC m=+6251.282337318" watchObservedRunningTime="2026-09-18 23:34:32.426820107 +0000 UTC m=+6251.283475758"
Sep 18 23:34:33.613414 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:33.613384    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:34:33.613832 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:34:33.613602    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:34:34.095755 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:34.095725    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tgn5c_070d51d0-7645-40a7-a7e8-9d1a9376b87d/dns/0.log"
Sep 18 23:34:34.115509 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:34.115482    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tgn5c_070d51d0-7645-40a7-a7e8-9d1a9376b87d/kube-rbac-proxy/0.log"
Sep 18 23:34:34.160801 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:34.160782    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lq478_bc73221c-de83-4fd3-b462-8644f373ae8a/dns-node-resolver/0.log"
Sep 18 23:34:34.745255 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:34.745224    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qs2dt_be981ec5-26f8-4362-8377-90002db40a57/node-ca/0.log"
Sep 18 23:34:35.613681 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:35.613655    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:34:35.613867 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:34:35.613843    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:34:35.872011 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:35.871904    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/0.log"
Sep 18 23:34:35.875675 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:35.875650    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/1.log"
Sep 18 23:34:36.424118 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:36.424084    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6ln68_edd0a4cb-7f0e-4d61-85e1-72fe73a17b2d/serve-healthcheck-canary/0.log"
Sep 18 23:34:36.929751 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:36.929717    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-ws6nf_211fed90-33df-4a21-af33-482fd0873638/insights-operator/0.log"
Sep 18 23:34:36.932602 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:36.932579    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-ws6nf_211fed90-33df-4a21-af33-482fd0873638/insights-operator/1.log"
Sep 18 23:34:36.953448 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:36.953429    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6jk4g_933fc10b-a510-420e-8cae-3a99f8061cd5/kube-rbac-proxy/0.log"
Sep 18 23:34:36.977887 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:36.977867    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6jk4g_933fc10b-a510-420e-8cae-3a99f8061cd5/exporter/0.log"
Sep 18 23:34:37.005222 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:37.005199    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6jk4g_933fc10b-a510-420e-8cae-3a99f8061cd5/extractor/0.log"
Sep 18 23:34:38.421581 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:38.421554    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-6wckh"
Sep 18 23:34:40.372001 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:40.371957    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7fdf8f455-wdj86_4a5ea3c8-b9e6-43cb-a3a9-670e5d2a0b0f/manager/0.log"
Sep 18 23:34:41.548026 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:41.547983    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/0.log"
Sep 18 23:34:41.552162 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:41.552140    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/1.log"
Sep 18 23:34:42.713012 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:42.712977    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/0.log"
Sep 18 23:34:42.715045 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:42.715021    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-s8kvl_f1ccfe42-36a8-4352-8ea6-9017406392bf/istio-proxy/1.log"
Sep 18 23:34:44.613718 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:44.613691    2564 scope.go:117] "RemoveContainer" containerID="fa54fc0a6873a9a5f9fa78ea096c6b00891c2878fd62f8a2fd4a9a75c730e306"
Sep 18 23:34:44.614085 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:34:44.613863    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk_e2e-test-flow-control-smoke-1e98f46d(cea0a11c-8723-4ee5-b1c7-974bb48f243b)\"" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-c6579d6db-cxcmk" podUID="cea0a11c-8723-4ee5-b1c7-974bb48f243b"
Sep 18 23:34:47.334226 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:47.334182    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-kq9nw_126f7e2c-cb3f-4301-9820-305e42c92369/migrator/0.log"
Sep 18 23:34:47.375550 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:47.375525    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-kq9nw_126f7e2c-cb3f-4301-9820-305e42c92369/graceful-termination/0.log"
Sep 18 23:34:47.613580 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:47.613497    2564 scope.go:117] "RemoveContainer" containerID="7142f31493b2f08d226491a63393644bd161f32497f53cd2d932dbd729efb1e7"
Sep 18 23:34:47.613748 ip-10-0-130-79 kubenswrapper[2564]: E0918 23:34:47.613727    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864)\"" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr" podUID="9b30dbc3-fb8a-439e-98c4-73c6cd5f7864"
Sep 18 23:34:48.946507 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:48.946478    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2979h_a4f9c0dc-d777-4cc6-a84b-1563af98dd71/kube-multus/0.log"
Sep 18 23:34:49.009944 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:49.009922    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqzs_5456bfbd-26d0-451a-beac-e2567ac818e7/kube-multus-additional-cni-plugins/0.log"
Sep 18 23:34:49.031761 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:49.031722    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqzs_5456bfbd-26d0-451a-beac-e2567ac818e7/egress-router-binary-copy/0.log"
Sep 18 23:34:49.054316 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:49.054291    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqzs_5456bfbd-26d0-451a-beac-e2567ac818e7/cni-plugins/0.log"
Sep 18 23:34:49.074297 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:49.074258    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqzs_5456bfbd-26d0-451a-beac-e2567ac818e7/bond-cni-plugin/0.log"
Sep 18 23:34:49.095069 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:49.095036    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqzs_5456bfbd-26d0-451a-beac-e2567ac818e7/routeoverride-cni/0.log"
Sep 18 23:34:49.117938 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:49.117915    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqzs_5456bfbd-26d0-451a-beac-e2567ac818e7/whereabouts-cni-bincopy/0.log"
Sep 18 23:34:49.138013 ip-10-0-130-79 kubenswrapper[2564]: I0918 23:34:49.137992    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqzs_5456bfbd-26d0-451a-beac-e2567ac818e7/whereabouts-cni/0.log"